The Risks of Recourse in Binary Classification

TL;DR

The paper investigates whether algorithmic recourse explanations improve classification accuracy at the population level. It introduces a general learning-theoretic framework that compares risk under the original distribution $P$ to a recourse-adjusted distribution $Q_f$, distinguishing compliant and defiant responses to recourse, and extends results to probabilistic and surrogate losses as well as strategic deployment. The authors prove that recourse frequently increases risk, including for Bayes-optimal and near-optimal classifiers, and they analyze how deployment incentives can mitigate or amplify these effects; empirical experiments on synthetic and real data corroborate the theoretical findings. The work highlights that current recourse notions are not reliably beneficial, motivating alternatives such as contestability and further refinements of counterfactual mechanisms to avoid systemic harm.

Abstract

Algorithmic recourse provides explanations that help users overturn an unfavorable decision by a machine learning system. But so far very little attention has been paid to whether providing recourse is beneficial or not. We introduce an abstract learning-theoretic framework that compares the risks (i.e., expected losses) for classification with and without algorithmic recourse. This allows us to answer the question of when providing recourse is beneficial or harmful at the population level. Surprisingly, we find that there are many plausible scenarios in which providing recourse turns out to be harmful, because it pushes users to regions of higher class uncertainty and therefore leads to more mistakes. We further study whether the party deploying the classifier has an incentive to strategize in anticipation of having to provide recourse, and we find that sometimes they do, to the detriment of their users. Providing algorithmic recourse may therefore also be harmful at the systemic level. We confirm our theoretical findings in experiments on simulated and real-world data. All in all, we conclude that the current concept of algorithmic recourse is not reliably beneficial, and therefore requires rethinking.

Figures (8)

• Figure 1: Left panel: Initial situation, the ML model classifies individual with starting features $x_0$ either negatively (in blue) or positively (in red). Its risk is denoted by $R_P(f)$. Points classified negatively are given the opportunity to move to the decision boundary (yellow dotted arrows). Right panel: The points close enough to the boundary accept recourse and move towards the decision boundary. The risk with recourse, $R_{Q_f}(f)$, is then higher, because at the decision boundary the uncertainty about the true class is maximal, and the points that accepted recourse are now more likely to be misclassified.
• Figure 2: Left: Bayes classifier, original predictions; Right: predictions after providing recourse in the compliant case.
• Figure 3: From left to right: Moons, Circles and Gaussian datasets. The left image for each shows the classifications with gradient boosted trees; the right image shows the effect of giving recourse.
• Figure 4: From left to right: Moons, Circles and Gaussian datasets. The left image for each shows the risk difference when $p \in [0,1]$; the right image shows the risk difference when $\sigma^2 \in [10^{-3}, 10^1]$ on a logarithmic scale
• Figure 5: Left figure: Linear classifier with a shaded area to indicate where recourse is accepted. Right figure: The same linear classifier but shifted towards the right in such a way that the effective decision boundary is the original decision boundary.

Theorems & Definitions (23)

• Theorem 1: Bayes-Optimal Classifier Risk Increase
• Theorem 2: Probabilistic Classifier Risk Increase, $0/1$ loss
• Theorem 3: Probabilistic Classifier Risk Increase, Surrogate Loss
• Theorem 4: Strategizing in the Defiant Case
• Definition 1
• Theorem 5: Strategizing in the Compliant Case
• Lemma 6
• proof
• Lemma 7
• proof