The Impact of Differential Feature Under-reporting on Algorithmic Fairness

TL;DR

This paper tackles fairness in public-sector risk models under differential feature under-reporting, a form of MNAR bias where data are more complete for populations relying more on public services. It introduces a tractable model with $X = Z \odot \xi$, analyzes how under-reporting induces attenuation and weight-shifting in linear predictors, and defines excess selection rates to quantify disparities. The authors show that standard missing-data methods generally fail to mitigate the resulting unfairness and propose two targeted strategies: model estimation with an augmented loss and group-dependent optimal prediction imputation, together with a mechanism to estimate under-reporting rates via PU-learning. Empirical results on semi-synthetic and real datasets demonstrate that under-reporting often increases disparities, while the proposed methods substantially reduce excess selection rates with minimal loss in predictive accuracy, offering practical guidance for fairer public-sector algorithms.

Abstract

Predictive risk models in the public sector are commonly developed using administrative data that is more complete for subpopulations that more greatly rely on public services. In the United States, for instance, information on health care utilization is routinely available to government agencies for individuals supported by Medicaid and Medicare, but not for the privately insured. Critiques of public sector algorithms have identified such differential feature under-reporting as a driver of disparities in algorithmic decision-making. Yet this form of data bias remains understudied from a technical viewpoint. While prior work has examined the fairness impacts of additive feature noise and features that are clearly marked as missing, the setting of data missingness absent indicators (i.e. differential feature under-reporting) has been lacking in research attention. In this work, we present an analytically tractable model of differential feature under-reporting which we then use to characterize the impact of this kind of data bias on algorithmic fairness. We demonstrate how standard missing data methods typically fail to mitigate bias in this setting, and propose a new set of methods specifically tailored to differential feature under-reporting. Our results show that, in real world data settings, under-reporting typically leads to increasing disparities. The proposed solution methods show success in mitigating increases in unfairness.

Figures (14)

• Figure 1: We study a prediction model on feature vectors with differential under-reporting $X$ where true outcomes $Y$ are a function of the latent 'true' features $Z$. Missingness $\xi$ is influenced by group membership $G$. We consider both cases in which feature distributions vary by group membership and cases with $G\perp Z$. In our setting, missingness indicators $\xi$ are unobserved and group membership $G$ is only used for model evaluation and not as a feature. The graph reflects the dependencies at prediction time.
• Figure 2: Group-wise excess selection rates using the ACS Income dataset. Each panel represents a feature that has been corrupted by under-reporting in independent runs of the experiment. Black curves show performance when omitting the entire feature column. Results are averaged over 50 runs on the test set. Shaded areas correspond to one standard deviation in each direction of the mean.
• Figure 3: Excess selection rates of group Other (i.e. not African-American) (left columns), parameter estimates (middle column), and test set $R^2$ (right columns) when under-reporting is injected into 'priors count' in group Other using the COMPAS dataset and synthetic two-year recidivism outcomes. In (a), the model is trained and evaluated using the under-reported feature. For (b), we first train a multiple imputation model and then train and evaluate the prediction model using probabilistic imputations. For (c), the model is trained on only rows without 0-entries in 'priors count' and evaluated on the under-reported data. In (d), we train with group-dependent augmented loss and use group-dependent optimal imputation values for prediction. Results are reported as averages over 30 runs. Shaded areas correspond to one standard deviation. The solid dots in the middle column correspond to true parameters. Note that in order to preserve readability, parameter estimates are only displayed for continuous features. Figure \ref{['fig:r_sq_facet']} provides an overlay plot of the rightmost column for easy comparison.
• Figure 4: Selection rate fractions of different models. On the left, the results are displayed for the sub-population of Black individuals. On the right, the results are displayed for the sub-population that is insured through Medicaid. The selection rate of the whole population is considered to be 10% or lower which reflects a realistic range for predictive risk modeling.
• Figure E.1: Excess selection rate of racial group Other (i.e. not African-American) at different selection rates of the whole population with synthetic two-year recidivism outcomes using the COMPAS dataset. Each panel represents a feature that has been corrupted by under-reporting in independent runs of the experiment. Feature under-reporting is added to the Other group with 0-90% missing in 10 percentage point increments. The black curves show performance when excluding the whole feature column from modeling. Results are reported as averages over 30 runs on a test set. Shaded areas correspond to one standard deviation in each direction of the mean.

Theorems & Definitions (13)

• Definition 1: Excess selection rate due to under-reporting
• Definition 2: Excess selection rate due to under-reporting, independent case
• Lemma 3: Attenuation bias
• Proposition 4: Properties of $\hat{\beta}_1$
• Proposition 5: Properties of $\hat{\beta}_k$
• Proposition 6
• Corollary 7
• Lemma 8: Augmented loss
• Lemma 9: Optimal prediction imputation value
• Proposition B.1