Measuring Individual User Fairness with User Similarity and Effectiveness Disparity

TL;DR

Measuring Individual User Fairness with User Similarity and Effectiveness Disparity introduces Pairwise User unFairness (PUF), a joint measure that weights disparity in effectiveness by user similarity to capture true individual fairness in RSs. PUF is defined as the weighted mean pairwise difference in effectiveness across all user pairs, $PUF = \frac{2}{m(m-1)} \sum_{u \in U}\sum_{u' \in U\setminus\{u\}} sim(u,u') \times |S(u)-S(u')|$, with $sim(u,u')\in[0,1]$ and $S(\cdot)\in[0,1]$, enabling applicability across different ranking metrics. Empirical evaluation across four real-world datasets (Lastfm, QK-video, ML-10M, ML-20M) and seven rankers demonstrates that PUF robustly captures both user similarity and effectiveness disparities, outperforming existing measures in sensitivity and providing faster computation. The work recommends adopting PUF for attribute-free RS fairness assessments and provides public code for replication.

Abstract

Individual user fairness is commonly understood as treating similar users similarly. In Recommender Systems (RSs), several evaluation measures exist for quantifying individual user fairness. These measures evaluate fairness via either: (i) the disparity in RS effectiveness scores regardless of user similarity, or (ii) the disparity in items recommended to similar users regardless of item relevance. Both disparity in recommendation effectiveness and user similarity are very important in fairness, yet no existing individual user fairness measure simultaneously accounts for both. In brief, current user fairness evaluation measures implement a largely incomplete definition of fairness. To fill this gap, we present Pairwise User unFairness (PUF), a novel evaluation measure of individual user fairness that considers both effectiveness disparity and user similarity. PUF is the only measure that can express this important distinction. We empirically validate that PUF does this consistently across 4 datasets and 7 rankers, and robustly when varying user similarity or effectiveness. In contrast, all other measures are either almost insensitive to effectiveness disparity or completely insensitive to user similarity. We contribute the first RS evaluation measure to reliably capture both user similarity and effectiveness in individual user fairness. Our code: https://github.com/theresiavr/PUF-individual-user-fairness-recsys.

Figures (4)

• Figure 1: Kendall's $\tau$ correlation between all measures (Eff, Fair, and PUFs).
• Figure 2: Effectiveness (Eff) and fairness (Fair) scores of QK-video and ML-20M, when artificially varying % of users with all irrelevant items (zero relevance), and the rest of the users receiving all relevant items. All PUF variants overlap. Gini is missing points at 100% users with zero relevance as it is undefined when each user has zero Eff scores.
• Figure 3: Artificially varying the skewness of the user similarity distribution for QK-video and ML-20M. Vertical grey lines denote the skewness corresponding to $sim_{Jacc}$ observed in the dataset. The distribution skewness differs across datasets.
• Figure 4: Artificially varying the % of users with zero relevance for QK-video and ML-20M. Lower Eff score difference is assigned to user pairs with higher similarity (MostFair), and to lower similarity (MostUnfair). Both UF overlap.