Counterfactual Fairness in Text Classification through Robustness

TL;DR

The paper tackles fairness in text toxicity classification by focusing on counterfactual changes to identity tokens. It introduces Counterfactual Token Fairness (CTF) and develops three training strategies—blindness, counterfactual augmentation, and counterfactual logit pairing (CLP)—to enforce robustness to identity substitutions. Empirical results show blindness and CLP can achieve near-zero CTF gaps with minimal or favorable impact on accuracy, while highlighting tradeoffs with group fairness. The work provides a practical framework for token-level fairness in NLP and motivates future research on asymmetric counterfactuals and richer counterfactual generation.

Abstract

In this paper, we study counterfactual fairness in text classification, which asks the question: How would the prediction change if the sensitive attribute referenced in the example were different? Toxicity classifiers demonstrate a counterfactual fairness issue by predicting that "Some people are gay" is toxic while "Some people are straight" is nontoxic. We offer a metric, counterfactual token fairness (CTF), for measuring this particular form of fairness in text classifiers, and describe its relationship with group fairness. Further, we offer three approaches, blindness, counterfactual augmentation, and counterfactual logit pairing (CLP), for optimizing counterfactual token fairness during training, bridging the robustness and fairness literature. Empirically, we find that blindness and CLP address counterfactual token fairness. The methods do not harm classifier performance, and have varying tradeoffs with group fairness. These approaches, both for measurement and optimization, provide a new path forward for addressing fairness concerns in text classification.

Figures (1)

• Figure 1: Plot of the average CTF gap along with the TPR and TNR over examples that contain identity terms.

Theorems & Definitions (2)

• Definition 1
• Definition 2