A Simple yet Effective Framework for Active Learning to Rank

TL;DR

The paper addresses active learning for web-scale ranking in Baidu’s search, proposing two complementary criteria, Ranking Entropy ($RE$) and Prediction Variance ($PV$), to select informative query–webpage pairs for annotation. By combining them as $f(q)=RE(q)+\alpha\cdot PV(q)$, the approach mitigates individual biases and improves ranking performance measured by $DCG_4$, achieving $\Delta DCG_4=1.38\%$ offline and $0.05\%$ to $0.35\%$ online under the same labeling budget. Empirical analyses demonstrate that $RE$ tends to favor low-frequency queries while $PV$ emphasizes high-frequency ones, and that PV correlates with true label variance and information gain; the combined strategy yields more valid training pairs and better top-K rankings. The results substantiate the practical impact of efficient active labeling for LTR in a real-world, trillions-scale web environment, with implications for scalable retrieval systems and data-efficient ranking improvements.

Abstract

While China has become the biggest online market in the world with around 1 billion internet users, Baidu runs the world largest Chinese search engine serving more than hundreds of millions of daily active users and responding billions queries per day. To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results. Among these components used in Baidu search, learning to rank (LTR) plays a critical role and we need to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of Activ Learning to Rank (active LTR) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion -- Ranking Entropy (RE) characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely Prediction Variances (PV) that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (i.e., the relative improvement ΔDCG4=1.38%) with the same budgeted labeling efforts.

Figures (7)

• Figure 1: An overview of Baidu search system with the proposed active learning process. While the search engine records every search query from users and stores them in Historical Search Queries, it periodically picks up the NEW queries that appeared within the last ONE month for annotation and re-trains LTR models with annotated data.
• Figure 2: Correlation Studies and Empirical Observations on Criteria based on 1000 Queries ($N=1000$). Best $DCG_4$ refer to the estimate of the upper bound of $DCG_4$, where we use the human-annotated ground truth labels to compute the $DCG_4$ score for every query.
• Figure 3: The distribution of 1,000 selected queries using different criteria. LV stands for label variance, PV for prediction variance and RE for ranking entropy.
• Figure 4: The distribution of labels over categories. We use 1,000 selected queries and the corresponding webpages to obtain the statistics.
• Figure 5: The relative improvements of $DCG_4$ (i.e., $\Delta DCG_4$) and $R_{01}$ (i.e., $\Delta R_{01}$) compared with using random selection in each active learning cycle with the same budget. Top: the base set is composed of 100 queries. Bottom: the base set is composed of 500 queries.