Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing

TL;DR

The paper develops a general framework for counterfactually fair reinforcement learning in sequential decision making, with a focus on healthcare applications. It proves that the optimal CF policy is stationary in stationary CMDPs and introduces a sequential data preprocessing algorithm under additive noise to estimate counterfactual states and rewards, enabling offline RL for CF policies. The authors provide theoretical guarantees on regret and unfairness control and demonstrate performance on synthetic, semi-synthetic, and real PowerED data, achieving substantially reduced CF unfairness with minimal loss in population-level value. The work advances CF in dynamic settings, offering a practical pipeline for fair, data-efficient RL in healthcare interventions like opioid misuse counseling.

Abstract

When applied in healthcare, reinforcement learning (RL) seeks to dynamically match the right interventions to subjects to maximize population benefit. However, the learned policy may disproportionately allocate efficacious actions to one subpopulation, creating or exacerbating disparities in other socioeconomically-disadvantaged subgroups. These biases tend to occur in multi-stage decision making and can be self-perpetuating, which if unaccounted for could cause serious unintended consequences that limit access to care or treatment benefit. Counterfactual fairness (CF) offers a promising statistical tool grounded in causal inference to formulate and study fairness. In this paper, we propose a general framework for fair sequential decision making. We theoretically characterize the optimal CF policy and prove its stationarity, which greatly simplifies the search for optimal CF policies by leveraging existing RL algorithms. The theory also motivates a sequential data preprocessing algorithm to achieve CF decision making under an additive noise assumption. We prove and then validate our policy learning approach in controlling unfairness and attaining optimal value through simulations. Analysis of a digital health dataset designed to reduce opioid misuse shows that our proposal greatly enhances fair access to counseling.

Figures (5)

• Figure 1: A simple example of SCM and counterfactual inference.
• Figure 2: Causal DAG of CMDP.
• Figure 3: Causal DAG of contextual bandit (one-stage).
• Figure 4: Comparisons under the linear setting (top row) and non-linear setting (bottom role): a,d) CF metric versus sample size, b,e) cumulative discounted reward versus CF metric (multiple sample sizes) with $\delta=1$, c,f) CF metric versus $\delta$. All the results are aggregated over 100 random seeds. The shaded area represents the $95\%$ CI.
• Figure 5: Semi-synthetic experiments: CF metric of different approaches versus $\eta$ for different sensitive attributes: education, sex, ethnicity. All the results are aggregated over 100 random seeds. The shaded area is the $95\%$ CI.

Theorems & Definitions (10)

• Definition 1: CMDP
• Remark 1
• Definition 2: Counterfactual Fairness in Contextual Bandit
• Definition 3: Counterfactual Fairness in CMDP
• Remark 2
• Theorem 1: Counterfactual augmentation
• Theorem 2: Optimality in stationary CMDPs
• Remark 3
• Theorem 3: Regret Bound
• Theorem 4: Unfairness Control