Monotone Individual Fairness

TL;DR

A characterization for auditing schemes that are capable of aggregating feedback from any number of auditors, using a rich class the authors term monotone aggregation functions is proved, practically reducing the analysis of auditing for individual fairness by multiple auditors to that of auditing by (instance-specific) single auditors.

Abstract

We revisit the problem of online learning with individual fairness, where an online learner strives to maximize predictive accuracy while ensuring that similar individuals are treated similarly. We first extend the frameworks of Gillen et al. (2018); Bechavod et al. (2020), which rely on feedback from human auditors regarding fairness violations, as we consider auditing schemes that are capable of aggregating feedback from any number of auditors, using a rich class we term monotone aggregation functions. We then prove a characterization for such auditing schemes, practically reducing the analysis of auditing for individual fairness by multiple auditors to that of auditing by (instance-specific) single auditors. Using our generalized framework, we present an oracle-efficient algorithm achieving an upper bound frontier of $(\mathcal{O}(T^{1/2+2b}),\mathcal{O}(T^{3/4-b}))$ respectively for regret, number of fairness violations, for $0\leq b \leq 1/4$. We then study an online classification setting where label feedback is available for positively-predicted individuals only, and present an oracle-efficient algorithm achieving an upper bound frontier of $(\mathcal{O}(T^{2/3+2b}),\mathcal{O}(T^{5/6-b}))$ for regret, number of fairness violations, for $0\leq b \leq 1/6$. In both settings, our algorithms improve on the best known bounds for oracle-efficient algorithms. Furthermore, our algorithms offer significant improvements in computational efficiency, greatly reducing the number of required calls to an (offline) optimization oracle per round, to $\tilde{\mathcal{O}}(α^{-2})$ in the full information setting, and $\tilde{\mathcal{O}}(α^{-2} + k^2T^{1/3})$ in the partial information setting, where $α$ is the sensitivity for reporting fairness violations, and $k$ is the number of individuals in a round.

Figures (3)

• Figure 1: An illustration of the possible objection profiles of a panel of $m=5$ auditors $j^1, \dots, j^5$ with respect to the predictions made on a pair $(x,x')\in\mathcal{X}^2$. In the figure, $d^1,\dots,d^5$ are shortened notation for $d^1(x,x'),\dots,d^5(x,x')$. In the example, the most strict auditor with respect to $(x,x')$ is $j^2$, whereas $j^4$ is the most lenient. Note that whenever an auditor objects to a prediction on $(x,x')$, all auditors to her left on the diagram object as well --- hence, in fact, only $m+1$ objection profiles (instead of $2^m$) are possible. The colored areas correspond to the prediction differences on $(x,x')$ that induce an aggregate violation (in this example, we set $\alpha = 0$), according to two (monotone) aggregation functions: $\bar{f}^1 = 1$ if and only if $j^3$ objects or at least an 80% majority of objections is reached (yellow), and $\bar{f}^2 = 1$ if and only if both $j^1$ and $j^5$ object (hatched). We can see that the "pivot" auditors are $j^3$ in the first case, and $j^1$ in the second.
• Figure 2: A low-dimensional geometric illustration of the constraints and feedback structure in our model. The polygon represents $\Delta\mathcal{H}^{fair}(\Psi^t)$. Each of the $k^2T$ halfspaces that define the polygon corresponds to an individual fairness constraint binding over a pair of individuals. At round $t$, only $k^2$ of the $k^2T$ constraints that define the polygon are active. Note that this polygon is implicit --- the learner does not know it at any point. Rather, under our feedback model, we are only guaranteed that if, at any round $t$, one or more of the $k^2$ active constraints at that round is violated by at least $\alpha$, the direction (see Definition \ref{['def:violation']}) to one of the halspaces representing these constraints, but not the actual distance from it, is revealed to the learner.
• Figure 3: An illustration of constraint elicitation (left) and evaluation (right). Since at round $t$, with high probability, $\forall i\in[k]: \left\vert\pi^t(\bar{x}^{t,i}) - \tilde{\pi}^t(\bar{x}^{t,i})\right\vert \leq \frac{\epsilon}{8}$, we know that in order for $\pi^t$ to have an $\alpha$-violation on a pair of individuals from $\bar{x}^t$, $\tilde{\pi}^t$ must have a $(\alpha-\frac{\epsilon}{2})$-violation on that pair as well. Note that the right side of the diagram is counterfactual --- the learner has no access to $\pi^t$ and cannot query the panel using it. However, as we see in the argument above, querying $\tilde{\pi}^t$ for $(\alpha-\frac{\epsilon}{2})$-violations is sufficient for the task of upper bounding the number of $\alpha$-violations by $\pi^t$.

Theorems & Definitions (37)

• Definition 2.1: Fairness violation
• Definition 2.2: Auditor
• Remark 2.3
• Definition 2.4: Auditing scheme
• Definition 2.5: Independent aggregation functions
• Definition 2.6: Aggregation order
• Definition 2.7: Monotone aggregation functions
• Definition 2.8: Monotone auditing scheme
• Lemma 2.9
• proof : Proof of Lemma \ref{['lem:monotone']}