Uncovering Fairness through Data Complexity as an Early Indicator

TL;DR

This study investigates whether differences in subgroup data complexity between privileged and unprivileged groups can serve as early indicators of algorithmic unfairness. It combines a large-scale synthetic bias framework (73 datasets) with three classifiers (LR, DT, KN) and association-rule mining to link complexity gaps to SP, EO, and PP, complemented by validation on 30 real-world datasets. The authors identify consistent patterns—especially class-imbalance (C2), boundary-overlap (N1), and local density (density)—that correlate with fairness violations and demonstrate that complexity-difference signals can guide pre-processing and model choices to mitigate bias. The findings advocate for routine complexity audits in the ML pipeline, offering data-centric indicators that help practitioners anticipate and address fairness challenges in practice. $CMD = |complexity extsubscript{privileged} - complexity extsubscript{unprivileged}|$ and the fair interval is $[-0.1, 0.1]$.

Abstract

Fairness constitutes a concern within machine learning (ML) applications. Currently, there is no study on how disparities in classification complexity between privileged and unprivileged groups could influence the fairness of solutions, which serves as a preliminary indicator of potential unfairness. In this work, we investigate this gap, specifically, we focus on synthetic datasets designed to capture a variety of biases ranging from historical bias to measurement and representational bias to evaluate how various complexity metrics differences correlate with group fairness metrics. We then apply association rule mining to identify patterns that link disproportionate complexity differences between groups with fairness-related outcomes, offering data-centric indicators to guide bias mitigation. Our findings are also validated by their application in real-world problems, providing evidence that quantifying group-wise classification complexity can uncover early indicators of potential fairness challenges. This investigation helps practitioners to proactively address bias in classification tasks.

Figures (8)

• Figure 1: Distribution of absolute differences in various complexity metrics when comparing privileged and unprivileged groups. Each metric (labeled along the horizontal axis) is represented by a separate distribution illustrating how its absolute difference values are spread. The vertical axis reflects the magnitude of these differences. The unbiased dataset marked as a red star. Each complexity metric absolute difference exhibits unique range coverage and variability patterns.
• Figure 2: Distribution of complexity metrics absolute differences by each bias scenario. Each subplot shows boxplot distributions for a specific bias scenario, with red stars indicating the baseline (S1A) values. S1B and S1C show no differences in complexity. S1E, S2A and S3A show similar differences distributions and the rest of the scenarios present different unique distributions of the complexity metrics.
• Figure 3: Evolution of complexity metrics difference by bias parameter. Each subplot represents a bias scenario in which each line represent a complexity metric. The horizontal axis represents the index of the array of values (Values column on Table \ref{['tab:syn_ds']}) of the parameter (P value on Table \ref{['tab:syn_ds']}) controlling the bias magnitude on each scenario. The vertical axes represent the magnitude of the differences in complexity. The evolution of complexity metrics across bias parameters reveals that while some metrics consistently increase with bias, others fluctuate based on specific scenario characteristics.
• Figure 5: Two-dimensional MDS visualization of synthetic datasets complexity metrics absolute differences colored by bias scenario. The baseline scenario (S1A) is marked with a star. Point sizes correspond to bias parameter magnitude, with larger points indicating stronger bias. Datasets that appear close to each other in the plot are more similar, while datasets that appear far apart are more dissimilar. S1D, S1F and S4A show the biggest differences. S1E, S2A and S3 show almost identical distributions.
• Figure 6: Distribution of fairness metrics results (Statistical Parity (SP), Equal Opportunity (EO), and Predictive Parity (PP)) computed using different methods (k-Nearest Neighbors (KN), Decision Trees (DT), and Logistic Regression (LR)) by each bias scenario. The horizontal axis show fairness metrics results for different ML methods represented by a separate distribution illustrating how its values spread. The vertical axis reflects the magnitude of fairness metrics. The gray band representing acceptable fairness bounds (±0.1). S1A, S1B and S1C are confirmed to have fair results. The biggest differences can be observed in SP in the rest of the scenarios.