Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systems

TL;DR

The theoretical results show that for users, up-weighting more value-faithful and less noisy behaviors leads to higher utility, while for producers, up-weighting more value-faithful and strategy-robust behaviors leads to higher welfare.

Abstract

Many recommender systems optimize a linear weighting of different user behaviors, such as clicks, likes, and shares. We analyze the optimal choice of weights from the perspectives of both users and content producers who strategically respond to the weights. We consider three aspects of each potential behavior: value-faithfulness (how well a behavior indicates whether the user values the content), strategy-robustness (how hard it is for producers to manipulate the behavior), and noisiness (how much estimation error there is in predicting the behavior). Our theoretical results show that for users, up-weighting more value-faithful and less noisy behaviors leads to higher utility, while for producers, up-weighting more value-faithful and strategy-robust behaviors leads to higher welfare (and the impact of noise is non-monotonic). Finally, we apply our framework to design weights on Facebook, using a large-scale dataset of approximately 70 million URLs shared on Facebook. Strikingly, we find that our user-optimal weight vector (a) delivers higher user value than a vector not accounting for variance; (b) also enhances broader societal outcomes, reducing misinformation and raising the quality of the URL domains, outcomes that were not directly targeted in our theoretical framework.

Figures (5)

• Figure 1: The optimal user weight vector as a function of value-faithfulness and variance. The optimal weight $\mathbf{w}_2$ on the second behavior increases as its value-faithfulness increases and decreases as its variance increases.
• Figure 2: The user-optimal and producer-optimal weight vector as a function of three aspects of behavior: value-faithfulness, variance, and strategy-robustness.
• Figure 3: The effects of the user-optimal, VF-only, and Facebook weight vector. The user-optimal and VF-only weight vectors were estimated 12 times, each based on data from a different month in 2017. The Facebook weight vector was reported to be unchangeduntil 2018 merrill2021five, and therefore, is constant across the 12 months. The estimated user-optimal and VF-only weight vector from one month of data (e.g. January 2017) was used to rank the next month of URLs (e.g. February 2017). The figure shows the weight vectors (left), the amount of misinformation (center) and domain quality (right) of the top URLs, averaged across months. All error bars represent 95% bootstrap confidence intervals.
• Figure 4: The optimal weight vector as a function of value-faithfulness and variance in the homogeneous setting. The default parameters for the simulations are $\mu_{\texttt{click}} = \mu_{\texttt{rec}} = 1$ and $\Sigma_{11} = \Sigma_{22} = 2$. The left figure is generated by plotting the optimal weight vector as $\mu_{\texttt{rec}}$ increases (and consequently, when value-faithfulness increases), and the right figure is generated by increasing $\Sigma_{22}$. The confidence bands show 95% confidence intervals based on 100 simulations of the data.
• Figure 5: A comparison of the empirical and theoretical optimal weight vector in the homogeneous and heterogeneous setting. In both settings, the variance on click is $\Sigma_{11} = 0.1$ and the variance on recommend is $\Sigma_{22} = 2$. In the homogeneous setting, $\mu_{\texttt{click}} = 1$ and $\mu_{\texttt{rec}} = 3$. In the heterogeneous setting, $\alpha_{\texttt{click}} = 0$, $\beta_{\texttt{click}} = 2$, $\alpha_{\texttt{rec}} = 2$, and $\beta_{\texttt{click}} = 4$. Thus, the mean predictions across items are the same in both the homogeneous and heterogeneous setting: $\mu_{\texttt{click}} = (\mu_{\texttt{i, click}})/n_{+}$ and $\mu_{\texttt{rec}} = (\mu_{\texttt{i, rec}})/n_{+}$. However, while the theory optimal weight vector and empirical optimal weight vector closely match in the homogeneous setting, they have a distinct gap in the heterogeneous setting.

Theorems & Definitions (9)

• Definition 3.1: Equilibrium
• Theorem 4.1
• Corollary 4.1
• Theorem 4.2
• Proposition 5.1: Equilibrium
• Corollary 5.1
• Proposition 5.2
• Theorem 5.1
• Lemma A.1