Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)

Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dawei Yin

TL;DR

This work tackles core web-LTR challenges by proposing GS$^2$P, a Generative Semi-Supervised Pre-trained ranking model. It combines semi-supervised pseudo-label generation, a denoising self-attentive representation learner, and an over-parameterized MLP ranker built on Random Fourier Features to operate effectively in the interpolation regime. Core contributions include co-training-based pseudo-labeling for unlabeled data, a Transformer-based denoising autoencoder that yields robust query–document representations, and an over-parameterized ranker that jointly optimizes discriminative and generative losses with a cross-validated feature dimension $N$. Extensive offline evaluations on Web30K and Baidu data, plus online A/B testing in a large-scale search engine, show consistent gains over strong baselines, especially under low-label regimes, demonstrating practical impact for real-world retrieval systems.

Abstract

Learning to rank (LTR) is widely employed in web searches to prioritize pertinent webpages from retrieved content based on input queries. However, traditional LTR models encounter two principal obstacles that lead to suboptimal performance: (1) the lack of well-annotated query-webpage pairs with ranking scores covering a diverse range of search query popularities, which hampers their ability to address queries across the popularity spectrum, and (2) inadequately trained models that fail to induce generalized representations for LTR, resulting in overfitting. To address these challenges, we propose a \emph{\uline{G}enerative \uline{S}emi-\uline{S}upervised \uline{P}re-trained} (GS2P) LTR model. We conduct extensive offline experiments on both a publicly available dataset and a real-world dataset collected from a large-scale search engine. Furthermore, we deploy GS2P in a large-scale web search engine with realistic traffic, where we observe significant improvements in the real-world application.

Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)

TL;DR

This work tackles core web-LTR challenges by proposing GSP, a Generative Semi-Supervised Pre-trained ranking model. It combines semi-supervised pseudo-label generation, a denoising self-attentive representation learner, and an over-parameterized MLP ranker built on Random Fourier Features to operate effectively in the interpolation regime. Core contributions include co-training-based pseudo-labeling for unlabeled data, a Transformer-based denoising autoencoder that yields robust query–document representations, and an over-parameterized ranker that jointly optimizes discriminative and generative losses with a cross-validated feature dimension . Extensive offline evaluations on Web30K and Baidu data, plus online A/B testing in a large-scale search engine, show consistent gains over strong baselines, especially under low-label regimes, demonstrating practical impact for real-world retrieval systems.

Abstract

Learning to rank (LTR) is widely employed in web searches to prioritize pertinent webpages from retrieved content based on input queries. However, traditional LTR models encounter two principal obstacles that lead to suboptimal performance: (1) the lack of well-annotated query-webpage pairs with ranking scores covering a diverse range of search query popularities, which hampers their ability to address queries across the popularity spectrum, and (2) inadequately trained models that fail to induce generalized representations for LTR, resulting in overfitting. To address these challenges, we propose a \emph{\uline{G}enerative \uline{S}emi-\uline{S}upervised \uline{P}re-trained} (GS2P) LTR model. We conduct extensive offline experiments on both a publicly available dataset and a real-world dataset collected from a large-scale search engine. Furthermore, we deploy GS2P in a large-scale web search engine with realistic traffic, where we observe significant improvements in the real-world application.
Paper Structure (12 sections, 2 figures, 3 tables)

This paper contains 12 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Semi-supervised Pseudo-Label Generation
  5. Self-attentive Representation Learning via Denoising Autoencoding
  6. LTR via Over-parameterized MLP
  7. Experiments
  8. Experimental Setup
  9. Offline Experimental Results
  10. Online Evaluation
  11. Related Works
  12. Conclusion

Figures (2)

  • Figure 1: The framework of GS$^2$P.
  • Figure 2: A/B test results of GS$^2$P and the legacy system for 7 days (t-test with $p < 0.05$ over the baseline).