Learning-to-Rank with Nested Feedback

TL;DR

This work addresses learning-to-rank in nested-feed interfaces where a 2nd-level feedback stream can inform the 1st-level ranking. It proposes augmenting the 1st-level relevance with L2 signals via $\widetilde{R}(u,a_i) = R_A(u,a_i) + \sum_{j=1}^{m} R_B(u,b_{ij})$, and optimizes $Q = \mathbb{E}_{u} \sum_i \widetilde{R}(u,a_i)$ under the position-based model; because the L2 ranking is fixed, its DCG contribution is treated as a summed quantity. It transforms historical logs to $$(u,A,\widetilde{Y})$$ with $\widetilde{y}_i = \sum_j y_{ij}$ to feed standard LTR methods, and uses YetiRank/CatBoost with Bayesian tuning for offline evaluation. Empirical validation on a ShareChat dataset shows that incorporating L2 feedback (through $S_2$ and $S_3$) reduces DCG loss and improves online metrics such as engagements, L1→L2 transitions, and retention, with the sum-based $S_3$ yielding the strongest gains.

Abstract

Many platforms on the web present ranked lists of content to users, typically optimized for engagement-, satisfaction- or retention- driven metrics. Advances in the Learning-to-Rank (LTR) research literature have enabled rapid growth in this application area. Several popular interfaces now include nested lists, where users can enter a 2nd-level feed via any given 1st-level item. Naturally, this has implications for evaluation metrics, objective functions, and the ranking policies we wish to learn. We propose a theoretically grounded method to incorporate 2nd-level feedback into any 1st-level ranking model. Online experiments on a large-scale recommendation system confirm our theoretical findings.

Figures (1)

• Figure 1: A nested-feed interface. When receiving feedback on item $n.k$ in the 2nd-level feed (denoted $\star$), the 1st-level item $n$ should be attributed as well.