Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability and Multigroup Fairness in Ranking with Uncertain Predictions

Siddartha Devic, Aleksandra Korolova, David Kempe, Vatsal Sharan

TL;DR

This work introduces ranking functions that translate uncertainty-aware multiclass predictions into distributions over rankings, and rigorously analyzes their stability and fairness properties. It proves that randomness is essential for achieving meaningful stability and shows that Uncertainty Aware (UA) rankings are anonymous and stable, while also enjoying multigroup fairness guarantees when paired with multiaccurate or multicalibrated predictors. The results establish an interpolation between individual and group fairness through multicalibration, and they quantify the tradeoffs between stability, fairness, and utility, including a practical mixing approach between UA and utility-optimal rankings. Empirical evaluations on real datasets demonstrate that UA rankings are robust to predictive noise and to some extent superior in utility compared to baselines, supporting their practical deployment for fairer, more stable ranking under uncertainty.

Abstract

Rankings are ubiquitous across many applications, from search engines to hiring committees. In practice, many rankings are derived from the output of predictors. However, when predictors trained for classification tasks have intrinsic uncertainty, it is not obvious how this uncertainty should be represented in the derived rankings. Our work considers ranking functions: maps from individual predictions for a classification task to distributions over rankings. We focus on two aspects of ranking functions: stability to perturbations in predictions and fairness towards both individuals and subgroups. Not only is stability an important requirement for its own sake, but -- as we show -- it composes harmoniously with individual fairness in the sense of Dwork et al. (2012). While deterministic ranking functions cannot be stable aside from trivial scenarios, we show that the recently proposed uncertainty aware (UA) ranking functions of Singh et al. (2021) are stable. Our main result is that UA rankings also achieve multigroup fairness through successful composition with multiaccurate or multicalibrated predictors. Our work demonstrates that UA rankings naturally interpolate between group and individual level fairness guarantees, while simultaneously satisfying stability guarantees important whenever machine-learned predictions are used.

Stability and Multigroup Fairness in Ranking with Uncertain Predictions

TL;DR

This work introduces ranking functions that translate uncertainty-aware multiclass predictions into distributions over rankings, and rigorously analyzes their stability and fairness properties. It proves that randomness is essential for achieving meaningful stability and shows that Uncertainty Aware (UA) rankings are anonymous and stable, while also enjoying multigroup fairness guarantees when paired with multiaccurate or multicalibrated predictors. The results establish an interpolation between individual and group fairness through multicalibration, and they quantify the tradeoffs between stability, fairness, and utility, including a practical mixing approach between UA and utility-optimal rankings. Empirical evaluations on real datasets demonstrate that UA rankings are robust to predictive noise and to some extent superior in utility compared to baselines, supporting their practical deployment for fairer, more stable ranking under uncertainty.

Abstract

Rankings are ubiquitous across many applications, from search engines to hiring committees. In practice, many rankings are derived from the output of predictors. However, when predictors trained for classification tasks have intrinsic uncertainty, it is not obvious how this uncertainty should be represented in the derived rankings. Our work considers ranking functions: maps from individual predictions for a classification task to distributions over rankings. We focus on two aspects of ranking functions: stability to perturbations in predictions and fairness towards both individuals and subgroups. Not only is stability an important requirement for its own sake, but -- as we show -- it composes harmoniously with individual fairness in the sense of Dwork et al. (2012). While deterministic ranking functions cannot be stable aside from trivial scenarios, we show that the recently proposed uncertainty aware (UA) ranking functions of Singh et al. (2021) are stable. Our main result is that UA rankings also achieve multigroup fairness through successful composition with multiaccurate or multicalibrated predictors. Our work demonstrates that UA rankings naturally interpolate between group and individual level fairness guarantees, while simultaneously satisfying stability guarantees important whenever machine-learned predictions are used.
Paper Structure (28 sections, 15 theorems, 36 equations, 1 figure, 2 tables)

This paper contains 28 sections, 15 theorems, 36 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Anonymity.
  3. Stability.
  4. Our Contributions
  5. Related Work
  6. Fairness in Ranking.
  7. Uncertainty in Rankings.
  8. Calibration and Ranking.
  9. Stability in Rankings.
  10. Notation and Preliminaries
  11. Predictors
  12. Rankings and Ranking Functions
  13. Desiderata of Ranking Functions
  14. Predictions and Rankings
  15. Consequences of Stability
  16. ...and 13 more sections

Key Result

Proposition 1

Let $f: \mathcal{X} \to \Delta_L$ be a $(\beta, d)$-individually fair predictor, and $r: \mathcal{P}_{n,L} \to \mathcal{M}^{n\times n}_{\text{DS}}$ an anonymous and $\gamma$-stable ranking function. Given a data set of individuals $(x_i)_{i \in [n]}$ and their associated predictions $P = (f(x_i))_{i

Figures (1)

  • Figure 1: An overview of our setting, using students being ranked by an employer for potential interviews. Observations (a), given by the students' resumés and coding abilities, are fed into a machine learning algorithm which produces distributions (b) over the relevance classes {irrelevant, suitable, extremely relevant} for each candidate. Then, a ranking function takes as input these uncertain predictions to produce a distribution (c) over rankings of the three candidates. Although it may appear that $x_3$ is the most qualified or relevant, due to inherent uncertainty in observations, a ranking function may place $x_1$ or $x_2$ at rank one with non-zero probability (c).

Theorems & Definitions (37)

  • Definition 1
  • Definition 2: Anonymity
  • Definition 3: Stability
  • Proposition 1
  • proof
  • Corollary 1
  • Definition 4: Determinism
  • Proposition 2
  • proof
  • Definition 5: Uncertainty Awareness singh2021fairness
  • ...and 27 more