Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Recourse Costs from Pairwise Feature Comparisons

Kaivalya Rawal, Himabindu Lakkaraju

TL;DR

This work addresses learning numeric per-feature modification costs for actionable recourse in black-box models by applying the Bradley-Terry model to map pairwise comparisons into feature strengths $β_f$, with costs given by Cost(f) = -β_f. It uses MAP estimation (via Gibbs sampling) to infer these costs from human pairwise comparisons, initially assuming exhaustive feature pairs and then extending to non-exhaustive data by using comparisons between entire recourses. The key contribution is showing that non-exhaustive data, including recourse-vs-recourse comparisons, can recover the full feature-cost vector and thus guide efficient recourse generation without explicit numeric elicitation. This approach enables practical, interpretable recourse in high-stakes settings and reduces the burden on users, with publicly available code supporting implementation.

Abstract

This paper presents a novel technique for incorporating user input when learning and inferring user preferences. When trying to provide users of black-box machine learning models with actionable recourse, we often wish to incorporate their personal preferences about the ease of modifying each individual feature. These recourse finding algorithms usually require an exhaustive set of tuples associating each feature to its cost of modification. Since it is hard to obtain such costs by directly surveying humans, in this paper, we propose the use of the Bradley-Terry model to automatically infer feature-wise costs using non-exhaustive human comparison surveys. We propose that users only provide inputs comparing entire recourses, with all candidate feature modifications, determining which recourses are easier to implement relative to others, without explicit quantification of their costs. We demonstrate the efficient learning of individual feature costs using MAP estimates, and show that these non-exhaustive human surveys, which do not necessarily contain data for each feature pair comparison, are sufficient to learn an exhaustive set of feature costs, where each feature is associated with a modification cost.

Learning Recourse Costs from Pairwise Feature Comparisons

TL;DR

This work addresses learning numeric per-feature modification costs for actionable recourse in black-box models by applying the Bradley-Terry model to map pairwise comparisons into feature strengths , with costs given by Cost(f) = -β_f. It uses MAP estimation (via Gibbs sampling) to infer these costs from human pairwise comparisons, initially assuming exhaustive feature pairs and then extending to non-exhaustive data by using comparisons between entire recourses. The key contribution is showing that non-exhaustive data, including recourse-vs-recourse comparisons, can recover the full feature-cost vector and thus guide efficient recourse generation without explicit numeric elicitation. This approach enables practical, interpretable recourse in high-stakes settings and reduces the burden on users, with publicly available code supporting implementation.

Abstract

This paper presents a novel technique for incorporating user input when learning and inferring user preferences. When trying to provide users of black-box machine learning models with actionable recourse, we often wish to incorporate their personal preferences about the ease of modifying each individual feature. These recourse finding algorithms usually require an exhaustive set of tuples associating each feature to its cost of modification. Since it is hard to obtain such costs by directly surveying humans, in this paper, we propose the use of the Bradley-Terry model to automatically infer feature-wise costs using non-exhaustive human comparison surveys. We propose that users only provide inputs comparing entire recourses, with all candidate feature modifications, determining which recourses are easier to implement relative to others, without explicit quantification of their costs. We demonstrate the efficient learning of individual feature costs using MAP estimates, and show that these non-exhaustive human surveys, which do not necessarily contain data for each feature pair comparison, are sufficient to learn an exhaustive set of feature costs, where each feature is associated with a modification cost.
Paper Structure (5 sections, 1 theorem, 2 equations, 5 figures)

This paper contains 5 sections, 1 theorem, 2 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Simulations
  4. Conclusion
  5. Appendix

Key Result

Theorem 2.1

Given an order over feature costs, without their exact values, not all possible recourses can be disambiguated.

Figures (5)

  • Figure 1: An example of a data instance that is denied a loan by a black box model, and two potential recourses that, if implemented, would get the loan application approved. Numeric costs on the ease of modification for each of the four features for Recourse 1 and Recourse 2 are essential to be able to determine which recourse is better for the user overall.
  • Figure 2: Mean Squared Error of final estimated Bradley-Terry strength parameters (w.r.t initial) for increasing sizes of survey data (increasing $\frac{Total Number of Comparisons}{|\mathcal{F}|}$). Four different simulations, each with a different size of $|\mathcal{F}|$ ranging from 5 to 20.
  • Figure 3: Running times for the experiment in Figure 2.
  • Figure 4: Mean Squared Error of the final estimated Bradley-Terry strength parameters (w.r.t initial) of a 20 feature model, when survey data consists of comparisons of recourses instead of single features. Recourse sizes simulated range from 1 to 6.
  • Figure 5: Running times for the experiment in Figure 4.

Theorems & Definitions (2)

  • Theorem 2.1
  • proof : Proof