Measuring Fairness in Large-Scale Recommendation Systems with Missing Labels

TL;DR

Measuring fairness in large-scale recommender systems is challenged by pervasive missing labels; the paper formalizes Ranking-based Equal Opportunity (REO) with $U_k = \mathbb{P}(R=1 \mid Y=1, S=s_k)$ and a global penalty $\Delta_{\mathrm{REO}} = \mathrm{std}(U_1, \dots, U_K)/\mathrm{mean}(U_1, \dots, U_K)$. To achieve identifiability, it introduces a small randomized traffic with activation probability $p_{act}$ that yields observable quantities from both random and default traffic, enabling consistent estimation via $\hat{P}_k$, $\hat{Q}_k$, $\hat{U}_k$, $\widehat{\Delta U_k}$ and $\widehat{\Delta_{REO}}$. The authors provide theoretical error bounds, a delta-method-based confidence interval, and fast significance tests for A/B comparisons, along with practical guidance on random-traffic volume and a scalable fairness monitoring pipeline. A real-world TikTok fairness dataset collected in Japan under random traffic demonstrates the approach's practicality and the necessity of random probing for reliable fairness estimates in production systems.

Abstract

In large-scale recommendation systems, the vast array of items makes it infeasible to obtain accurate user preferences for each product, resulting in a common issue of missing labels. Typically, only items previously recommended to users have associated ground truth data. Although there is extensive research on fairness concerning fully observed user-item interactions, the challenge of fairness in scenarios with missing labels remains underexplored. Previous methods often treat these samples missing labels as negative, which can significantly deviate from the ground truth fairness metrics. Our study addresses this gap by proposing a novel method employing a small randomized traffic to estimate fairness metrics accurately. We present theoretical bounds for the estimation error of our fairness metric and support our findings with empirical evidence on real data. Our numerical experiments on synthetic and TikTok's real-world data validate our theory and show the efficiency and effectiveness of our novel methods. To the best of our knowledge, we are the first to emphasize the necessity of random traffic in dataset collection for recommendation fairness, the first to publish a fairness-related dataset from TikTok and to provide reliable estimates of fairness metrics in the context of large-scale recommendation systems with missing labels.

Figures (13)

• Figure 1: An illustration of Random Traffic through forced insertion and Default Traffic through the default recommendation strategy, which generates $\mathcal{D}_{rand}$ and $\mathcal{D}_{rec}$ respectively.
• Figure 2: An illustration of uniform sampling in Random Traffic $R(u,i)=1$ (resp. $0$) means $i$ is recommended (resp. not recommended) in the default traffic for a user request $u$.
• Figure 3: The mean squared error of fairness penalty on synthetical data with variable traffic size.
• Figure 4: Relative error of fairness penalty estimation as a function of dataset size.
• Figure 5: Variance of fairness penalty estimation as a function of dataset size.

Theorems & Definitions (21)

• Example 3.1
• Lemma 3.2: informal
• Theorem 3.3: informal
• Theorem 3.4: informal
• Definition C.1: Admissible set of unprobed dataset
• Definition C.2: Nontrivial recommended subset
• Lemma C.3
• proof
• Theorem C.4: formal version of \ref{['thm:identifiability']}
• proof