Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods

TL;DR

This paper investigates whether post-hoc feature attribution explanations for language models are fair across gender. It introduces a systematic evaluation across three NLP tasks and five LM families using six local attribution methods and a suite of metrics for faithfulness, robustness, and complexity. The main findings show significant gender disparities in explanation quality across most configurations, persisting even when models are trained from scratch on unbiased data; larger models and unbiased pretraining can mitigate some disparities but do not eliminate them. The authors discuss practical and regulatory implications, calling for auditing of explanation fairness and incorporating explanation-level fairness into policy and method development.

Abstract

While research on applications and evaluations of explanation methods continues to expand, fairness of the explanation methods concerning disparities in their performance across subgroups remains an often overlooked aspect. In this paper, we address this gap by showing that, across three tasks and five language models, widely used post-hoc feature attribution methods exhibit significant gender disparity with respect to their faithfulness, robustness, and complexity. These disparities persist even when the models are pre-trained or fine-tuned on particularly unbiased datasets, indicating that the disparities we observe are not merely consequences of biased training data. Our results highlight the importance of addressing disparities in explanations when developing and applying explainability methods, as these can lead to biased outcomes against certain subgroups, with particularly critical implications in high-stakes contexts. Furthermore, our findings underscore the importance of incorporating the fairness of explanations, alongside overall model fairness and explainability, as a requirement in regulatory frameworks.

Figures (7)

• Figure 1: Overview of our experimental pipeline, exemplified with the GECO dataset wilming2024gecobench. We begin by obtaining predictions for male/female sentence pairs. We then use feature attribution methods to explain the predictions and evaluate the explanations using various metrics. We finally analyze the distributions of evaluation scores per each metric for male and female sentences and observe if the evaluations differ significantly between the two genders, indicating gender bias and disparity in explanations.
• Figure 2: Box-plots of evaluation scores obtained over 5 runs for each using TinyBERT on GECO, Stereotypes, and COMPAS, including the runs not resulting in statistically significant disparity.
• Figure 3: Box-plots of the soft comprehensiveness and sufficiency metrics obtained over 5 runs for each using TinyBERT on GECO-ALL, Stereotypes, and COMPAS, including the runs not resulting in statistically significant disparity.
• Figure 4: Box-plots of evaluation scores obtained over 5 runs for each using BERT on our datasets, including the runs not resulting in statistically significant disparity.
• Figure 5: Box-plots of evaluation scores obtained over 5 runs for each using FairBERTa on our datasets, including the runs not resulting in statistically significant disparity.