Analyzing Fairness of Neural Network Prediction via Counterfactual Dataset Generation

TL;DR

This work introduces counterfactual dataset (CFD) generation to audit training-time label bias in neural networks. By combining a linear regression surrogate (training-stage) with neuron activation similarity (inference-stage), the method efficiently ranks training examples and flips up to a budget $m$ of labels to produce CFDs that change a target test prediction. Across seven fairness datasets and 1100+ test cases, CFDs are generated with high efficiency and markedly better fidelity than baselines, often finding a CFD in the first iteration. The findings reveal that small label biases can substantially influence individual predictions, offering an interpretable tool for diagnosing dataset bias and auditing model fairness in practice.

Abstract

Interpreting the inference-time behavior of deep neural networks remains a challenging problem. Existing approaches to counterfactual explanation typically ask: What is the closest alternative input that would alter the model's prediction in a desired way? In contrast, we explore counterfactual datasets. Rather than perturbing the input, our method efficiently finds the closest alternative training dataset, one that differs from the original dataset by changing a few labels. Training a new model on this altered dataset can then lead to a different prediction of a given test instance. This perspective provides a new way to assess fairness by directly analyzing the influence of label bias on training and inference. Our approach can be characterized as probing whether a given prediction depends on biased labels. Since exhaustively enumerating all possible alternate datasets is infeasible, we develop analysis techniques that trace how bias in the training data may propagate through the learning algorithm to the trained network. Our method heuristically ranks and modifies the labels of a bounded number of training examples to construct a counterfactual dataset, retrains the model, and checks whether its prediction on a chosen test case changes. We evaluate our approach on feedforward neural networks across over 1100 test cases from 7 widely-used fairness datasets. Results show that it modifies only a small subset of training labels, highlighting its ability to pinpoint the critical training examples that drive prediction changes. Finally, we demonstrate how our counterfactual datasets reveal connections between training examples and test cases, offering an interpretable way to probe dataset bias.

Figures (7)

• Figure 1: Our method to efficiently generate counterfactual dataset (CFD) $D'$ for an input $x$, based on the original dataset $D$, learning algorithm $\mathcal{L}$, and a bound $m$ on the number of training label flips.
• Figure 2: A neural network with ReLU activation for a given input $\mathbf{x}$. Active neurons are outlined in red.
• Figure B.1: Comparison of training examples selected by our method and baseline approaches for CFD generation. Each plot shows feature values of the test input alongside the chosen training examples.
• Figure B.2: Comparison of original $y$ logit values for successful cases for various datasets. Boxplots show quartiles (boxes) and min/max ranges (whiskers). Colors denote methods: blue for Our Method, orange for Random Sampling, and magenta for $L_2$ distance.
• Figure D.1: Venn diagrams showing overlap of CFDs found by Our Method (filled), Random Sampling, and $L_2$ Distance across datasets. Numbers inside each region indicate the total CFDs found, as reported in \ref{['tab:results']}.