GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

TL;DR

This work addresses how gender biases in pre-trained language models propagate into feature attributions produced by XAI methods. It introduces GECO, a gender-controlled dataset with three variants (M, F, NB) and two ground-truth sets (D_S and D_A), alongside GECOBench, a framework for evaluating explanation correctness via a ground-truth SAP and Mass Accuracy metric. The approach benchmarks multiple XAI methods (e.g., Saliency, Integrated Gradients, LIME, Kernel SHAP) on BERT under varying fine-tuning/retraining regimes, using a zero-shot baseline for comparison. Findings show residual biases in embeddings influence explanations and that targeted fine-tuning of embedding layers can improve explanation correctness, offering a principled path to assessing and mitigating bias in XAI for NLP.

Abstract

Large pre-trained language models have become a crucial backbone for many downstream tasks in natural language processing (NLP), and while they are trained on a plethora of data containing a variety of biases, such as gender biases, it has been shown that they can also inherit such biases in their weights, potentially affecting their prediction behavior. However, it is unclear to what extent these biases also affect feature attributions generated by applying "explainable artificial intelligence" (XAI) techniques, possibly in unfavorable ways. To systematically study this question, we create a gender-controlled text dataset, GECO, in which the alteration of grammatical gender forms induces class-specific words and provides ground truth feature attributions for gender classification tasks. This enables an objective evaluation of the correctness of XAI methods. We apply this dataset to the pre-trained BERT model, which we fine-tune to different degrees, to quantitatively measure how pre-training induces undesirable bias in feature attributions and to what extent fine-tuning can mitigate such explanation bias. To this extent, we provide GECOBench, a rigorous quantitative evaluation framework for benchmarking popular XAI methods. We show a clear dependency between explanation performance and the number of fine-tuned layers, where XAI methods are observed to benefit particularly from fine-tuning or complete retraining of embedding layers.

Figures (5)

• Figure 1: Overview of the benchmarking approach for evaluating the correctness of XAI methods. Starting from a clear definition of discriminative features inducing statistical associations between features/words and the prediction target, we specify ground truth explanations. With that, we craft a gender-focused dataset GECO, with text sourced from Wikipedia, by labeling and altering the grammatical gender of specific words. The resulting training and validation datasets are used to train the language model BERT. The test dataset, together with the trained model, serves as input to the XAI method, which outputs explanations for the test set. The word-based ground truth explanations, provided by the former labeling process, are then used to measure the correctness of each sentence's generated explanations using the Mass Accuracy metric arrasCLEVRXAIBenchmarkDataset2022clarkTetris2023clarkEXACTPlatformEmpirically2025.
• Figure 2: Explanation performance of different post-hoc XAI methods applied to language models that were adapted from BERT using five different transfer learning schemes. XAI evaluations were carried out on classified sentences in two gender-classification tasks, represented by datasets $\mathcal{D}_S$ and $\mathcal{D}_A$. The baseline performance for uniformly drawn random feature attributions is denoted by Uniform Random. Pattern Variant denotes a model- and pretraining-agnostic global explanation method. In (a), the relative change in explanation performance with respect to a zero-shot BERT model shows consistent changes for models with fine-tuned embeddings. In (b), fine-tuning or retraining of the embedding layers of BERT leads to consistent improvements in explanation correctness even when model performance is held constant for all models. Applying XAI methods to the OLA model leads to overall higher explanation performance, with InputXGradient becoming on par with Pattern Variant.
• Figure 3: Pearson correlation between tf-idf representation of words and the target for GECO. Here, we see the top ten words by correlation, and labeled words such as the pronouns he and she or his and her are consistently ranked highest in both datasets $\mathcal{D}_{A}$ and $\mathcal{D}_{S}$, indicating how the labeling introduced dependency between target and words.
• Figure 4: Model performance.
• Figure 5: Feature attributions by popular explanation methods for one sample sentence, broken down into input tokens as given to the respective model, with the ground truth manipulations highlighted in green. The majority of importance by many methods is correctly attributed to the word "she"; however, all tokenized words show non-zero attribution for multiple methods, including the character period ".".

Theorems & Definitions (1)

• Definition 3.1: Statistical Association Property (SAP)