Failing on Bias Mitigation: Investigating Why Predictive Models Struggle with Government Data

TL;DR

This study investigates why bias mitigation techniques fail when applied to government data, using Bristol crime-rate prediction as a controlled case study. It compares common regression models (MLP, DT, RF, GB, LR) under temporal and random splits, and analyzes fairness with discretized continuous sensitive features via Delta MAE, revealing persistent unfairness across race and religion groups despite mitigation. The authors identify data-centric root causes—distribution shifts, historical biases, and data-release delays—and demonstrate that intersectional fairness can reveal biases invisible to single-feature analyses. They argue for context-aware, interdisciplinary approaches and caution against assuming that standard mitigation methods will generalize across government data deployments, calling for dynamic models and closer collaboration with social scientists. These insights provide an early warning that biases in government datasets may persist even with conventional fairness interventions and guide more robust, context-sensitive AI governance in public services.

Abstract

The potential for bias and unfairness in AI-supporting government services raises ethical and legal concerns. Using crime rate prediction with the Bristol City Council data as a case study, we examine how these issues persist. Rather than auditing real-world deployed systems, our goal is to understand why widely adopted bias mitigation techniques often fail when applied to government data. Our findings reveal that bias mitigation approaches applied to government data are not always effective -- not because of flaws in model architecture or metric selection, but due to the inherent properties of the data itself. Through comparing a set of comprehensive models and fairness methods, our experiments consistently show that the mitigation efforts cannot overcome the embedded unfairness in the data -- further reinforcing that the origin of bias lies in the structure and history of government datasets. We then explore the reasons for the mitigation failures in predictive models on government data and highlight the potential sources of unfairness posed by data distribution shifts, the accumulation of historical bias, and delays in data release. We also discover the limitations of the blind spots in fairness analysis and bias mitigation methods when only targeting a single sensitive feature through a set of intersectional fairness experiments. Although this study is limited to one city, the findings are highly suggestive, which can contribute to an early warning that biases in government data may persist even with standard mitigation methods.

Figures (8)

• Figure 1: Single Feature Fairness Analysis. Each race feature is divided into low-high groups and tested on four different models. The difference between the High and Low groups of the same feature shows the unfairness when the model makes the prediction. The results in this figure are the average of 10 runs of experiments. The fairness analysis on religion features is shown in Figure \ref{['fig:single_rel']} in Appendix B.1.
• Figure 2: Scatter Plot of Data in Each Year. The data samples are visualized by t-SNE van2008visualizing. The feature distribution before (area in blue) and after 2020 (area in red) has significantly shifted in the feature space.
• Figure 3: The grey lines (left y-axis) represent individual Wards, while the blue line (right y-axis) highlights the Bristol Average. The purpose of using a dual-axis plot is to highlight the average trend in which (a) the requirement for free school meals suddenly increased after 2018 and (b) the crime rate showed a significant decrease in the COVID lockdown (2020/21) period.
• Figure 4: Single Feature Fairness Analysis on Religion Features.
• Figure 5: Single Feature Analysis without Sensitive Features Inputted during Training. There was still an unfairness in the sensitive feature groups in different machine learning models, even if the sensitive features were removed during training.