Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems

TL;DR

The paper addresses the misalignment between traditional ranking objectives (e.g., $NDCG$) and business goals in cascade ranking under varying data and model complexities. It proposes Adaptive Neural Ranking Framework (ARF), which uses a differentiable relaxation of Recall@$m@k$ via NeuralSort to create $L_{Relax}$ and couples it with a full-information objective $OPA$ through $L_{Global}$ in an uncertainty-weighted multi-task learning setup. The key contributions are: (1) a novel surrogate loss $L_{Relax}$ for directly optimizing $Recall@m@k$, (2) ARF that adaptively balances relaxed and full targets with $L_{Global}$, and (3) comprehensive offline and online experiments showing consistent recall improvements and substantial commercial value in a real-world cascade ranking system. The results demonstrate that adapting optimization targets to data complexity and stage capacity yields robust improvements across public and industrial datasets, with tangible gains in revenue and conversions when deployed online. This framework offers a practical path to align learning-to-rank with business goals in complex cascade architectures.

Abstract

Cascade ranking is widely used for large-scale top-k selection problems in online advertising and recommendation systems, and learning-to-rank is an important way to optimize the models in cascade ranking. Previous works on learning-to-rank usually focus on letting the model learn the complete order or top-k order, and adopt the corresponding rank metrics (e.g. OPA and NDCG@k) as optimization targets. However, these targets can not adapt to various cascade ranking scenarios with varying data complexities and model capabilities; and the existing metric-driven methods such as the Lambda framework can only optimize a rough upper bound of limited metrics, potentially resulting in sub-optimal and performance misalignment. To address these issues, we propose a novel perspective on optimizing cascade ranking systems by highlighting the adaptability of optimization targets to data complexities and model capabilities. Concretely, we employ multi-task learning to adaptively combine the optimization of relaxed and full targets, which refers to metrics Recall@m@k and OPA respectively. We also introduce permutation matrix to represent the rank metrics and employ differentiable sorting techniques to relax hard permutation matrix with controllable approximate error bound. This enables us to optimize both the relaxed and full targets directly and more appropriately. We named this method as Adaptive Neural Ranking Framework (abbreviated as ARF). Furthermore, we give a specific practice under ARF. We use the NeuralSort to obtain the relaxed permutation matrix and draw on the variant of the uncertainty weight method in multi-task learning to optimize the proposed losses jointly. Experiments on a total of 4 public and industrial benchmarks show the effectiveness and generalization of our method, and online experiment shows that our method has significant application value.

Figures (6)

• Figure 1: A classic cascade ranking architecture, which includes four stages: Matching, Pre-ranking, Ranking, and Re-ranking.
• Figure 2: The heatmap of the results on $Recall@m@k$ of $L^\lambda_{Recall@m@k}$ and $L_{Relax}@m@k$ under different $m$ and $k$, on Istella dataset.
• Figure 3: The heatmap of the results on $Recall@m@k$ of $L^\lambda_{Recall@m@k}$, on MSLR-WEB30K dataset.
• Figure 4: The heatmap of the results on $Recall@m@k$ of $L_{Relax}$, on MSLR-WEB30K dataset.
• Figure 5: The heatmap of the results on $Recall@m@k$ of $L^\lambda_{NDCG@k}$, on MSLR-WEB30K dataset.

Theorems & Definitions (2)

• Definition 1: the oracle condition for cascade ranking systems
• Definition 2: Stage Recall Complete Condition