Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)

Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, Dawei Yin

TL;DR

MPGraf tackles the challenge of unifying regression-based ranking and graph-based link prediction in web-scale LTR by introducing a modular Graphformer framework that can operate in stacked or parallelized configurations. It constructs graph data through Link Rippling, pre-trains GNN and Transformer modules on cross-domain LTR tasks, and applies surgical fine-tuning to adapt to target data while mitigating distribution shifts between graph and pair domains. Empirical results show MPGraf outperforms state-of-the-art baselines offline on public and commercial datasets and delivers meaningful online gains, especially for long-tail queries. This approach enables robust, end-to-end LTR at web scale by leveraging cross-domain pre-training, graph-aware representation learning, and careful parameter freezing during fine-tuning, with practical impact for search engine ranking systems.

Abstract

Both Transformer and Graph Neural Networks (GNNs) have been employed in the domain of learning to rank (LTR). However, these approaches adhere to two distinct yet complementary problem formulations: ranking score regression based on query-webpage pairs, and link prediction within query-webpage bipartite graphs, respectively. While it is possible to pre-train GNNs or Transformers on source datasets and subsequently fine-tune them on sparsely annotated LTR datasets, the distributional shifts between the pair-based and bipartite graph domains present significant challenges in integrating these heterogeneous models into a unified LTR framework at web scale. To address this, we introduce the novel MPGraf model, which leverages a modular and capsule-based pre-training strategy, aiming to cohesively integrate the regression capabilities of Transformers with the link prediction strengths of GNNs. We conduct extensive offline and online experiments to rigorously evaluate the performance of MPGraf.

Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)

TL;DR

MPGraf tackles the challenge of unifying regression-based ranking and graph-based link prediction in web-scale LTR by introducing a modular Graphformer framework that can operate in stacked or parallelized configurations. It constructs graph data through Link Rippling, pre-trains GNN and Transformer modules on cross-domain LTR tasks, and applies surgical fine-tuning to adapt to target data while mitigating distribution shifts between graph and pair domains. Empirical results show MPGraf outperforms state-of-the-art baselines offline on public and commercial datasets and delivers meaningful online gains, especially for long-tail queries. This approach enables robust, end-to-end LTR at web scale by leveraging cross-domain pre-training, graph-aware representation learning, and careful parameter freezing during fine-tuning, with practical impact for search engine ranking systems.

Abstract

Both Transformer and Graph Neural Networks (GNNs) have been employed in the domain of learning to rank (LTR). However, these approaches adhere to two distinct yet complementary problem formulations: ranking score regression based on query-webpage pairs, and link prediction within query-webpage bipartite graphs, respectively. While it is possible to pre-train GNNs or Transformers on source datasets and subsequently fine-tune them on sparsely annotated LTR datasets, the distributional shifts between the pair-based and bipartite graph domains present significant challenges in integrating these heterogeneous models into a unified LTR framework at web scale. To address this, we introduce the novel MPGraf model, which leverages a modular and capsule-based pre-training strategy, aiming to cohesively integrate the regression capabilities of Transformers with the link prediction strengths of GNNs. We conduct extensive offline and online experiments to rigorously evaluate the performance of MPGraf.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The Proposed Model
  3. Graph Construction with Link Rippling
  4. Representation Learning with Hybrid Graphformer
  5. Surgical Fine-tuning with Modular Composition
  6. Experiments
  7. Experimental Setup
  8. Offline Experimental Results
  9. Online Experimental Results
  10. Related Works
  11. Conclusion

Figures (2)

  • Figure 1: The framework of the proposed MPGraf.
  • Figure 2: Online comparative performance ($\Delta$NDCG$@{5}$) of MPGraf with various losses for 7 days (t-test with $p < 0.05$ over the baseline).