Balancing Domestic and Global Perspectives: Evaluating Dual-Calibration and LLM-Generated Nudges for Diverse News Recommendation

TL;DR

It is found that algorithmic nudges can successfully increase exposure and consumption diversity, while the impact of LLM-based presentation nudges varied, and longitudinal exposure to calibrated news may shift readers'reading habits to value a balanced news digest from both domestic and world articles.

Abstract

In this study, we applied the ``personalized diversity nudge framework'' with the goal of expanding user reading coverage in terms of news locality (i.e., domestic and world news). We designed a novel topic-locality dual calibration algorithmic nudge and a large language model-based news personalization presentation nudge, then launched a 5-week real-user study with 120 U.S. news readers on the news recommendation experiment platform POPROX. With user interaction logs and survey responses, we found that algorithmic nudges can successfully increase exposure and consumption diversity, while the impact of LLM-based presentation nudges varied. User-level topic interest is a strong predictor of user clicks, while highlighting the relevance of news articles to prior read articles outperforms generic topic-based and no personalization. We also demonstrate that longitudinal exposure to calibrated news may shift readers' reading habits to value a balanced news digest from both domestic and world articles. Our results provide direction for future work on nudging for diverse consumption in news recommendation systems.

Figures (6)

• Figure 1: Method overview of the calibration and personalized news preview generation workflow.
• Figure 2: Examples of rewrite strategies. Event-based prompts (top) integrate context from a previously clicked article with high cosine similarity. Topic-based prompts (bottom) fall back to incorporating top-3 interested topics. Prompt language shown is illustrative. Detailed prompts are included in Appendix Figure \ref{['fig:detail-prompt']}.
• Figure 3: Theta pair tuning metrics based on the one-month prior-experiment evaluation data. A higher NDCG@10 score is preferred, and a lower KL divergence score on theta topic and theta locality indicates better alignment. The red boxes indicate the final theta pair selection based on Eq. \ref{['eq:theta-definition']}.
• Figure 4: Forest plots of relative risk (log scale) from Eq. \ref{['eqn:formula1']} (red) and Eq. \ref{['eqn:formula2']} (blue). Smaller is better. Group comparisons relative to TC baseline with moderate KL-divergence (exposure intercept: 0.06 [0.04, 0.10], consumption intercept: 0.10 [0.06, 0.19]). Both treatments significantly reduce exposure divergence (DC: RR=0.03, p<.001; DC-NP: RR=0.08, p<.001) and consumption divergence (DC: RR=0.07, p<.001; DC-NP: RR=0.08, p<.001). Notably, there is no significant difference between DC and DC-NP consumption divergence. Week had a small, significant effect reducing exposure diversity (RR=0.71, p<.001) and consumption diversity (RR=0.68, p<.001) for the TC group, with minor deviation of mixed significance by treatment interactions.
• Figure 5: Event- and topic-based news rewrite prompt.