Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints

TL;DR

This work addresses fairness in crowdsourced label aggregation under Demographic Parity by deriving sharp, non-asymptotic bounds on the DP fairness gap for Majority Vote and Bayes aggregation, and by showing conditions under which the aggregated labels' DP gap converges to that of the ground truth. It introduces FairCrowd, a discrete, post-processing method that enforces explicit $b5$-fairness while preserving accuracy, and extends existing continuous fairness post-processing to discrete settings, including multiclass. The authors provide both theoretical insights and practical algorithms, demonstrating strong empirical performance on synthetic data and real crowdsourcing benchmarks such as Crowd Judgment and Jigsaw Toxicity. The work highlights that fairness-aware label aggregation can mitigate bias propagation in downstream models, offering a principled, efficient path to fair crowdsourced data pipelines. $$b5$$-fairness constraints and probabilistic post-processing yield robust improvements, especially in the small-crowd regime, and the approach unifies aggregation and fairness adjustment in a single framework.

Abstract

As acquiring reliable ground-truth labels is usually costly, or infeasible, crowdsourcing and aggregation of noisy human annotations is the typical resort. Aggregating subjective labels, though, may amplify individual biases, particularly regarding sensitive features, raising fairness concerns. Nonetheless, fairness in crowdsourced aggregation remains largely unexplored, with no existing convergence guarantees and only limited post-processing approaches for enforcing $\varepsilon$-fairness under demographic parity. We address this gap by analyzing the fairness s of crowdsourced aggregation methods within the $\varepsilon$-fairness framework, for Majority Vote and Optimal Bayesian aggregation. In the small-crowd regime, we derive an upper bound on the fairness gap of Majority Vote in terms of the fairness gaps of the individual annotators. We further show that the fairness gap of the aggregated consensus converges exponentially fast to that of the ground-truth under interpretable conditions. Since ground-truth itself may still be unfair, we generalize a state-of-the-art multiclass fairness post-processing algorithm from the continuous to the discrete setting, which enforces strict demographic parity constraints to any aggregation rule. Experiments on synthetic and real datasets demonstrate the effectiveness of our approach and corroborate the theoretical insights.

Figures (4)

• Figure 1: Overview of the CrowdFair algorithm.
• Figure 2: Convergence of the difference of demographic parity gap $\Delta_{\mathrm{DP}}(\hat{Y}^\phi)-\Delta_{\mathrm{DP}}(Y)$ according to the number of annotators $R$ for any $\phi\in \{\phi^*,\phi^{\mathrm{MV}},\phi^{DS}\}$ in the case where annotators (a) are globally competent $\mathcal{K}_{a} = \mathcal{U}([0.5\,\mathbb{I}_0(a) + 0.6\,\mathbb{I}_1(a), 1])$, (b) are globally adversarial $\mathcal{K}_{a} = \mathcal{U}([0.2 \mathbb{I}_0(a) + 0.1 \mathbb{I}_1(a), 0.6])$, (c) convey nearly no information.
• Figure 3: Performance comparison on the Synthetic dataset (a) and Crowd Judgement dataset (b). Solid lines correspond to our method (FC), while dashed lines indicate competing approaches. Shaded regions denote the variance across $10$ independent runs using different test set ($60$%).
• Figure 4: Performance comparison on the Jigsaw dataset. Solid lines correspond to our method (FC), while dashed lines indicate competing approaches. Shaded regions denote the variance across $10$ independent runs using different test set ($60$%).

Theorems & Definitions (50)

• Definition 2.1: Global DP Gap & $\varepsilon$-Fairness
• Lemma 2.2
• Theorem 3.1
• Proposition 3.2
• Lemma 3.3
• Theorem 3.4: Asymptotic Fairness Consistency
• Lemma 3.5
• Proposition 3.6
• Theorem 4.1
• Theorem 1.1