Reducing Population-level Inequality Can Improve Demographic Group Fairness: a Twitter Case Study

TL;DR

Empirically exploring the relationship between demographic-free inequality metrics -- such as the Gini coefficient -- and standard demographic bias metrics that measure group-wise model performance disparities specifically in the case of engagement inequality on Twitter finds that demographic-free impression inequality metrics are positively correlated with gender, race, and age disparities in the average case and weakly correlated with demographic bias in the worst case.

Abstract

Many existing fairness metrics measure group-wise demographic disparities in system behavior or model performance. Calculating these metrics requires access to demographic information, which, in industrial settings, is often unavailable. By contrast, economic inequality metrics, such as the Gini coefficient, require no demographic data to measure. However, reductions in economic inequality do not necessarily correspond to reductions in demographic disparities. In this paper, we empirically explore the relationship between demographic-free inequality metrics -- such as the Gini coefficient -- and standard demographic bias metrics that measure group-wise model performance disparities specifically in the case of engagement inequality on Twitter. We analyze tweets from 174K users over the duration of 2021 and find that demographic-free impression inequality metrics are positively correlated with gender, race, and age disparities in the average case, and weakly (but still positively) correlated with demographic bias in the worst case. We therefore recommend inequality metrics as a potentially useful proxy measure of average group-wise disparities, especially in cases where such disparities cannot be measured directly. Based on these results, we believe they can be used as part of broader efforts to improve fairness between demographic groups in scenarios like content recommendation on social media.

Figures (5)

• Figure 1: Toy example illustrating that changes to Gini need not correspond to changes in demographic disparities and vice versa.
• Figure 2: Demographic distribution of the tweet authors in our dataset.
• Figure 3: Spearman's correlation and p-values between marginal and intersectional bias metrics. The two rows of each subfigure correspond to Gini coefficient and T1PS, respectively. The columns are the demographic bias metrics, either marginal or intersectional (combination of two marginal metrics), with the suffix denoting whether it was a MAD or IMM metric.
• Figure 4: Daily tracking of Inequality Metrics (blue) and Marginal Bias Metrics (green) over 2021.
• Figure 5: Daily tracking of inequality metrics (blue) and intersectional bias metrics (green) over 2021.