USM: Unbiased Survey Modeling for Limiting Negative User Experiences in Recommendation Systems

TL;DR

The paper addresses the challenge of sparse negative feedback in recommendation systems by introducing in-feed surveys that collect user opinions and feed them into ranking. It proposes an end-to-end framework comprising an in-feed survey model enhanced with LHUC and SE modules and a survey-submit model to mitigate response bias via inverse propensity weighting, with equations for $P(like|ss)$ and the final item score $final\_s(item)$. The approach yields online benefits, including reductions in survey-related issues (e.g., sexual and inappropriate content reports) and in negative engagements (reports/dislikes), along with improvements in calibration and UAUC metrics, while long-term retention sees modest gains. Together, these contributions offer a practical pathway to more humane, user-aware recommendations by explicitly modeling and debiasing survey-based signals within deployment.

Abstract

Reducing negative user experiences is essential for the success of recommendation platforms. Exposing users to inappropriate content could not only adversely affect users' psychological well-beings, but also potentially drive users away from the platform, sabotaging the platform's long-term success. However, recommendation algorithms tend to weigh more heavily on positive feedback signals due to the scarcity of negative ones, which may result in the neglect of valuable negative user feedback. In this paper, we propose an approach aimed at limiting negative user experiences. Our method primarily relies on distributing in-feed surveys to the users, modeling the users' feedback collected from the survey, and integrating the model predictions into the recommendation system. We further enhance the baseline survey model by integrating the Learning Hidden Unit Contributions module and the Squeeze-and-Excitation module. In addition, we strive to resolve the problem of response Bias by applying a survey-submit model; The A/B testing results indicate a reduction in survey sexual rate and survey inappropriate rate, ranging from -1.44\% to -3.9\%. Additionally, we compared our methods against an online baseline that does not incorporate our approach. The results indicate that our approach significantly reduces the report rate and dislike rate by 1\% to 2.27\% compared to the baseline, confirming the effectiveness of our methods in enhancing user experience. After we launched the survey model based our approach on our platform, the model is able to bring reductions of 1.75\%, 2.57\%, 2.06\% on reports, dislikes, survey inappropriate rate, respectively.

Figures (5)

• Figure 1: In-feed survey example. (\ref{['fig:img1']}) illustrates the satisfaction survey designed to assess whether users appreciated the videos they recently viewed, offering three response options. The subfigures (\ref{['fig:img2']}) and (\ref{['fig:img3']}) illustrate the primary and secondary pages of the inappropriate survey. The primary page inquires whether the video is suitable on the TikTok platform; If users select "no" or "only for 18+," a secondary interface is activated, prompting them to specify their reasons for the selection.
• Figure 2: Illustration of the stages in a recommendation system. Recommendation systems can be roughly divided into four stages: recall, pre-rank, rank, and re-rank, ultimately selecting k videos (typically 8-10) as the final output for users.
• Figure 3: In-feed survey model structure. The model takes user and item features as inputs, along with a multi-head architecture where each head estimates the user's response for each specific option. We further enhanced the backbone by incorporating the LHUC and SE modules to improve feature cross-interaction and self-attention extraction.
• Figure 4: LHUC & SE module detail structure. (\ref{['fig:LHUC']}) The LHUC module utilizes the original input features to generate three output embeddings, which are multiplied with the outputs of the backbone FC layers;. (\ref{['fig:SE']}) The SE module performs global pooling for squeezing, followed by two FC layers to extract self-attention, and multiplies it with the input for excitation.
• Figure 5: Negtive Feedback Metric Reduction using Response Debias