Fair Active Ranking from Pairwise Preferences

TL;DR

This work investigates the problem of probably approximately correct and fair (PACF) ranking of items by adaptively evoking pairwise comparisons and proposes an objective function that generalizes the objective function of $\epsilon-Best-Ranking, proposed by Saha&Gopalan (2019).

Abstract

We investigate the problem of probably approximately correct and fair (PACF) ranking of items by adaptively evoking pairwise comparisons. Given a set of $n$ items that belong to disjoint groups, our goal is to find an $(ε, δ)$-PACF-Ranking according to a fair objective function that we propose. We assume access to an oracle, wherein, for each query, the learner can choose a pair of items and receive stochastic winner feedback from the oracle. Our proposed objective function asks to minimize the $\ell_q$ norm of the error of the groups, where the error of a group is the $\ell_p$ norm of the error of all the items within that group, for $p, q \geq 1$. This generalizes the objective function of $ε$-Best-Ranking, proposed by Saha & Gopalan (2019). By adopting our objective function, we gain the flexibility to explore fundamental fairness concepts like equal or proportionate errors within a unified framework. Adjusting parameters $p$ and $q$ allows tailoring to specific fairness preferences. We present both group-blind and group-aware algorithms and analyze their sample complexity. We provide matching lower bounds up to certain logarithmic factors for group-blind algorithms. For a restricted class of group-aware algorithms, we show that we can get reasonable lower bounds. We conduct comprehensive experiments on both real-world and synthetic datasets to complement our theoretical findings.

Figures (12)

• Figure 1: Example of different $\epsilon$-Best-Rankings.
• Figure 2: Group-Aware Ranking on German Credit with Age defining two groups age $<25$ (minority) and age $\ge 25$.
• Figure 3: Group-wise errors (for $n=25, p=q=1$) for $g-0$ (majority group) and $g-1$, $g-2$ (minority groups).
• Figure 4: Experiments on the real-world datasets for different values of $n$ (for $p = q = 1$).
• Figure 5: True scores of the synthetic datasets where in geo the scores decrease in a geometric progression, in arith the scores decrease in an arithmetic progression, in steps the scores decrease in an arithmetic progression but only for every 5 items, and for har the scores decrease in a harmonic progression. The colors of the bars represent the groups the items belong to.

Theorems & Definitions (37)

• Definition 1: $\epsilon$-Best- Fair-Ranking
• Theorem 1
• Remark
• Definition 2: $(\epsilon,\delta)$-PACF-Ranker
• Remark
• Theorem 1: sample complexity
• Remark
• proof : Proof Sketch
• Theorem 1: upper bound
• proof : Proof Sketch