Meta Learning to Rank for Sparsely Supervised Queries

TL;DR

This work proposes a novel meta-learning to rank framework which leverages fast learning and adaption capability of meta-learning and demonstrates that the proposed meta-learning approach can significantly enhance the performance of learning to rank models with sparsely labeled queries.

Abstract

Supervisory signals are a critical resource for training learning to rank models. In many real-world search and retrieval scenarios, these signals may not be readily available or could be costly to obtain for some queries. The examples include domains where labeling requires professional expertise, applications with strong privacy constraints, and user engagement information that are too scarce. We refer to these scenarios as sparsely supervised queries which pose significant challenges to traditional learning to rank models. In this work, we address sparsely supervised queries by proposing a novel meta learning to rank framework which leverages fast learning and adaption capability of meta-learning. The proposed approach accounts for the fact that different queries have different optimal parameters for their rankers, in contrast to traditional learning to rank models which only learn a global ranking model applied to all the queries. In consequence, the proposed method would yield significant advantages especially when new queries are of different characteristics with the training queries. Moreover, the proposed meta learning to rank framework is generic and flexible. We conduct a set of comprehensive experiments on both public datasets and a real-world e-commerce dataset. The results demonstrate that the proposed meta-learning approach can significantly enhance the performance of learning to rank models with sparsely labeled queries.

Figures (9)

• Figure 1: The architecture of the proposed meta learning to rank framework (MLTR) in the meta-training process
• Figure 2: Illustration of MLTR in meta training which optimizes for a representation $\theta$ that can quickly adapt to new queries. The orange dashed line represents the query-specific ranker initialized from the meta ranker and locally updated based on the training set data in meta training. The blue dashed line represents the direction of meta loss updates based on the updated query-specific ranker on test data in meta training. The purple solid line represents the global updates of the meta ranker based on the meta loss.
• Figure 3: Meta train/test evaluation on NDCG@10 of MLTR and other baselines with RankNet on the MSLR-10K dataset
• Figure 4: Comparison the performance of models without fine-tuning, using various loss functions and models, on the NDCG@10 metric of the entire test dataset $\mathcal{T}$ from four different public datasets. The symbol $\ddagger$ in the bar indicates a statistically significant improvement of MLTR without fine-tuning over the corresponding LTR models, as evidenced by a p-value $<$ 0.01 in a two-tailed t-test.
• Figure 5: Relative improvement experimental results of NDCG@10 from MLTR and LTR based on RankNet loss in variant sparsely labeled data setting on MSLR-10K dataset