Calibration-Disentangled Learning and Relevance-Prioritized Reranking for Calibrated Sequential Recommendation

TL;DR

LeapRec addresses the challenging trade-off between relevance and calibration in sequential recommendations by introducing a two-phase framework. It first trains with calibration-disentangled learning-to-rank to anticipate calibration effects, then applies a relevance-prioritized greedy reranking to place highly relevant items at the top while maintaining calibration. The approach yields superior trade-offs between accuracy (nDCG@10, HR@10) and sequential calibration (S_{KL}@10) across four real-world datasets, with ablations confirming the necessity of both components. The method is model-agnostic, efficient in reranking, and comes with open-source code, offering practical impact for systems requiring calibrated sequential recommendations.

Abstract

Calibrated recommendation, which aims to maintain personalized proportions of categories within recommendations, is crucial in practical scenarios since it enhances user satisfaction by reflecting diverse interests. However, achieving calibration in a sequential setting (i.e., calibrated sequential recommendation) is challenging due to the need to adapt to users' evolving preferences. Previous methods typically leverage reranking algorithms to calibrate recommendations after training a model without considering the effect of calibration and do not effectively tackle the conflict between relevance and calibration during the reranking process. In this work, we propose LeapRec (Calibration-Disentangled Learning and Relevance-Prioritized Reranking), a novel approach for the calibrated sequential recommendation that addresses these challenges. LeapRec consists of two phases, model training phase and reranking phase. In the training phase, a backbone model is trained using our proposed calibration-disentangled learning-to-rank loss, which optimizes personalized rankings while integrating calibration considerations. In the reranking phase, relevant items are prioritized at the top of the list, with items needed for calibration following later to address potential conflicts between relevance and calibration. Through extensive experiments on four real-world datasets, we show that LeapRec consistently outperforms previous methods in the calibrated sequential recommendation. Our code is available at https://github.com/jeon185/LeapRec.

Figures (9)

• Figure 1: KL divergence analysis of user-interacted category distributions over sequence intervals, employing a window size of 20 for each category distribution. This plot illustrates the shifts in category preferences over time in real-world datasets: ML-1M and Steam. Detailed data statistics are summarized in Table \ref{['table:datasets']}.
• Figure 2: Given user history (a), sequential calibration (d) applies more weight to recent category preference, in contrast to recommendation without calibration (b) and static calibration (c).
• Figure 3: LeapRec consists of two phases: (a) calibration-disentangled learning-to-rank and (b) relevance-prioritized reranking. In the first phase, a backbone model is trained to optimize personalized rankings, accommodating both with and without calibration considerations. In the second phase, items are greedily added to the recommendation list, where relevance is prioritized at higher ranks and calibration at lower ranks.
• Figure 4: Illustrative example of calibration-disentangled learning-to-rank. Initially, miscalibration scores are disentangled from relevance scores (a). Then, we train the model based on both $\mathcal{L}_{\mathit{BPR}}$ (b) and $\mathcal{L}_{\mathit{CD-BPR}}$ (c), resulting in a personalized pairwise ranking considering calibration (d).
• Figure 5: Trade-off comparison of LeapRec and baselines. The first row shows $nDCG@10$ vs. $\mathcal{S}_{KL}@10$, and the second row shows $HR@10$ vs. $\mathcal{S}_{KL}@10$. LeapRec outperforms the baselines on four real-world datasets, drawing better trade-off curves between accuracy and calibration.

Theorems & Definitions (1)

• Definition 1: Sequential miscalibration