Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects

Joel Persson, Jurriën Bakker, Dennis Bohle, Stefan Feuerriegel, Florian von Wangenheim

TL;DR

This work develops a unified statistical framework to detect and mitigate group bias in randomized experiments, and analyzes the economic implications of mitigating detected group bias for profit-maximizing personalized targeting.

Abstract

Heterogeneous treatment effects (HTEs) are increasingly estimated using machine learning models that produce highly personalized predictions of treatment effects. In practice, however, predicted treatment effects are rarely interpreted, reported, or audited at the individual level but, instead, are often aggregated to broader subgroups, such as demographic segments, risk strata, or markets. We show that such aggregation can induce systematic bias of the group-level causal effect: even when models for predicting the individual-level conditional average treatment effect (CATE) are correctly specified and trained on data from randomized experiments, aggregating the predicted CATEs up to the group level does not, in general, recover the corresponding group average treatment effect (GATE). We develop a unified statistical framework to detect and mitigate this form of group bias in randomized experiments. We first define group bias as the discrepancy between the model-implied and experimentally identified GATEs, derive an asymptotically normal estimator, and then provide a simple-to-implement statistical test. For mitigation, we propose a shrinkage-based bias-correction, and show that the theoretically optimal and empirically feasible solutions have closed-form expressions. The framework is fully general, imposes minimal assumptions, and only requires computing sample moments. We analyze the economic implications of mitigating detected group bias for profit-maximizing personalized targeting, thereby characterizing when bias correction alters targeting decisions and profits, and the trade-offs involved. Applications to large-scale experimental data at major digital platforms validate our theoretical results and demonstrate empirical performance.

Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects

TL;DR

This work develops a unified statistical framework to detect and mitigate group bias in randomized experiments, and analyzes the economic implications of mitigating detected group bias for profit-maximizing personalized targeting.

Abstract

Heterogeneous treatment effects (HTEs) are increasingly estimated using machine learning models that produce highly personalized predictions of treatment effects. In practice, however, predicted treatment effects are rarely interpreted, reported, or audited at the individual level but, instead, are often aggregated to broader subgroups, such as demographic segments, risk strata, or markets. We show that such aggregation can induce systematic bias of the group-level causal effect: even when models for predicting the individual-level conditional average treatment effect (CATE) are correctly specified and trained on data from randomized experiments, aggregating the predicted CATEs up to the group level does not, in general, recover the corresponding group average treatment effect (GATE). We develop a unified statistical framework to detect and mitigate this form of group bias in randomized experiments. We first define group bias as the discrepancy between the model-implied and experimentally identified GATEs, derive an asymptotically normal estimator, and then provide a simple-to-implement statistical test. For mitigation, we propose a shrinkage-based bias-correction, and show that the theoretically optimal and empirically feasible solutions have closed-form expressions. The framework is fully general, imposes minimal assumptions, and only requires computing sample moments. We analyze the economic implications of mitigating detected group bias for profit-maximizing personalized targeting, thereby characterizing when bias correction alters targeting decisions and profits, and the trade-offs involved. Applications to large-scale experimental data at major digital platforms validate our theoretical results and demonstrate empirical performance.
Paper Structure (47 sections, 4 theorems, 83 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 47 sections, 4 theorems, 83 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Algorithmic Bias in Machine Learning
  4. Bias in Causal Inference
  5. Bias Mitigation and Its Implications
  6. Motivating Example
  7. Framework
  8. Treatment Effect Estimands
  9. Group Bias
  10. Detection
  11. Estimation via Decomposition of Group Bias.
  12. Inference and Statistical Test.
  13. Mitigation
  14. Problem and Objective.
  15. Optimal Debiasing.
  16. ...and 32 more sections

Key Result

Proposition 1

Let $b_g = \tau_g^f - \tau_g$ be the group bias and let $\widehat{B}_g = \widehat{\tau}_g^f - \widehat{\tau}_g$, where $\widehat{\tau}_g^f$ and $\widehat{\tau}_g$ are sample estimators of $\tau_g^f$ and $\tau_g$, respectively, computed on a sample of effective size $N_g$. If $\widehat{\tau}_g^f$ and

Figures (9)

  • Figure 1: Simulation illustration of group bias in CATE predictions.
  • Figure 2: Experimentally estimated GATEs versus model-predicted GATEs in the hold-out data
  • Figure 3: Group bias estimates $\widehat{B}_g$ against relative group sample sizes $N_g / N$ in the hold-out data.
  • Figure 4: Bias Corrections and Profit Differences
  • Figure 5: Share of targeting decisions changed by debiasing as functions of its determinants.
  • ...and 4 more figures

Theorems & Definitions (8)

  • Definition 1: Collapsibility
  • Definition 2: Group Bias
  • Example 1
  • Example 2
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4