Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems

TL;DR

This work tackles the challenge of predicting long-term user retention in recommendation systems by proposing Retentive Relevance, a content-level survey that measures forward-looking intent to return. It validates the measure psychometrically and demonstrates its predictive superiority over traditional engagement signals and existing surveys through offline modeling and large-scale production deployments. A production-ready proxy and calibrated ranking integration enable real-time use, and online A/B experiments show consistent gains in retention, engagement, and content quality. The paper provides a scalable, user-centered framework that links content-level user perceptions to retention outcomes, with implications for responsible AI and platform growth.

Abstract

Recommendation systems have traditionally relied on short-term engagement signals, such as clicks and likes, to personalize content. However, these signals are often noisy, sparse, and insufficient for capturing long-term user satisfaction and retention. We introduce Retentive Relevance, a novel content-level survey-based feedback measure that directly assesses users' intent to return to the platform for similar content. Unlike other survey measures that focus on immediate satisfaction, Retentive Relevance targets forward-looking behavioral intentions, capturing longer term user intentions and providing a stronger predictor of retention. We validate Retentive Relevance using psychometric methods, establishing its convergent, discriminant, and behavioral validity. Through large-scale offline modeling, we show that Retentive Relevance significantly outperforms both engagement signals and other survey measures in predicting next-day retention, especially for users with limited historical engagement. We develop a production-ready proxy model that integrates Retentive Relevance into the final stage of a multi-stage ranking system on a social media platform. Calibrated score adjustments based on this model yield substantial improvements in engagement, and retention, while reducing exposure to low-quality content, as demonstrated by large-scale A/B experiments. This work provides the first empirically validated framework linking content-level user perceptions to retention outcomes in production systems. We offer a scalable, user-centered solution that advances both platform growth and user experience. Our work has broad implications for responsible AI development.

Figures (3)

• Figure 1: Schematic representation of the Retentive Relevance survey implementation. The interface maintains visual reference to the content being evaluated while capturing forward-looking behavioral intentions. "Platform" was replaced with the name of the social media app where the survey was deployed.
• Figure 2: Heatmap of Mutual Information Matrix shows that Retentive Relevance captures information about recommendation that is distinct from engagement signals.
• Figure 3: Feature importance analysis via SHAP values shows that Retentive Relevance significantly improves retention prediction—especially for low-signal users and with survey signals proving more predictive than engagement signals.