Controlled Personalization in Legacy Media Online Services: A Case Study in News Recommendation

TL;DR

This study investigates controlled personalization in a legacy news service by conducting a live A/B test at Aftenposten, where 20% personalization is added to the ranking while preserving editor-driven curation. The authors apply a multi-mmetric evaluation framework spanning user engagement (CTR, impressions, reading behavior) and journalistic values (diversity, coverage, popularity) to assess impacts beyond traditional CTR gains. Results show that modest personalization increases CTR, reduces navigation effort, and broadens content exposure while decreasing popularity bias, thereby aligning reader needs with editorial goals. The findings provide a practical blueprint for legacy media to deploy personalization in a way that respects journalistic values and fosters subscriber engagement, with implications for organizational governance and measurement practices.

Abstract

Personalized news recommendations have become a standard feature of large news aggregation services, optimizing user engagement through automated content selection. In contrast, legacy news media often approach personalization cautiously, striving to balance technological innovation with core editorial values. As a result, online platforms of traditional news outlets typically combine editorially curated content with algorithmically selected articles - a strategy we term controlled personalization. In this industry paper, we evaluate the effectiveness of controlled personalization through an A/B test conducted on the website of a major Norwegian legacy news organization. Our findings indicate that even a modest level of personalization yields substantial benefits. Specifically, we observe that users exposed to personalized content demonstrate higher click-through rates and reduced navigation effort, suggesting improved discovery of relevant content. Moreover, our analysis reveals that controlled personalization contributes to greater content diversity and catalog coverage and in addition reduces popularity bias. Overall, our results suggest that controlled personalization can successfully align user needs with editorial goals, offering a viable path for legacy media to adopt personalization technologies while upholding journalistic values.

Figures (11)

• Figure 1: Structure of Aftenposten's mobile front page.
• Figure 2: Probability density function of daily CTR.
• Figure 3: Probability density functions daily CTR per user activity level.
• Figure 4: Normalized distribution of impressions per section and variant.
• Figure 5: Normalized distribution of clicks per section and variant.