Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAT: Retrieval-Augmented Transformer for Click-Through Rate Prediction

Yushen Li, Jinpeng Wang, Tao Dai, Jieming Zhu, Jun Yuan, Rui Zhang, Shu-Tao Xia

TL;DR

This work tackles CTR prediction by incorporating cross-sample context through retrieval-augmented learning. It introduces Retrieval-Augmented Transformer (RAT), which retrieves similar samples via BM25 and processes a retrieval-augmented input with cascaded intra- and cross-sample attention in Transformer blocks, achieving efficient modeling with complexity $\mathcal{O}((K+1)^2+(F+1)^2)$. Experiments on ML-tag, KKBox, and Tmall show RAT outperforms traditional CTR models and prior RA methods (RIM, PET), with pronounced gains on long-tail data. The results demonstrate that leveraging cross-sample context and fine-grained interactions significantly improves robustness and accuracy in sparse feature regimes, offering practical benefits for real-world web applications.

Abstract

Predicting click-through rates (CTR) is a fundamental task for Web applications, where a key issue is to devise effective models for feature interactions. Current methodologies predominantly concentrate on modeling feature interactions within an individual sample, while overlooking the potential cross-sample relationships that can serve as a reference context to enhance the prediction. To make up for such deficiency, this paper develops a Retrieval-Augmented Transformer (RAT), aiming to acquire fine-grained feature interactions within and across samples. By retrieving similar samples, we construct augmented input for each target sample. We then build Transformer layers with cascaded attention to capture both intra- and cross-sample feature interactions, facilitating comprehensive reasoning for improved CTR prediction while retaining efficiency. Extensive experiments on real-world datasets substantiate the effectiveness of RAT and suggest its advantage in long-tail scenarios. The code has been open-sourced at \url{https://github.com/YushenLi807/WWW24-RAT}.

RAT: Retrieval-Augmented Transformer for Click-Through Rate Prediction

TL;DR

This work tackles CTR prediction by incorporating cross-sample context through retrieval-augmented learning. It introduces Retrieval-Augmented Transformer (RAT), which retrieves similar samples via BM25 and processes a retrieval-augmented input with cascaded intra- and cross-sample attention in Transformer blocks, achieving efficient modeling with complexity . Experiments on ML-tag, KKBox, and Tmall show RAT outperforms traditional CTR models and prior RA methods (RIM, PET), with pronounced gains on long-tail data. The results demonstrate that leveraging cross-sample context and fine-grained interactions significantly improves robustness and accuracy in sparse feature regimes, offering practical benefits for real-world web applications.

Abstract

Predicting click-through rates (CTR) is a fundamental task for Web applications, where a key issue is to devise effective models for feature interactions. Current methodologies predominantly concentrate on modeling feature interactions within an individual sample, while overlooking the potential cross-sample relationships that can serve as a reference context to enhance the prediction. To make up for such deficiency, this paper develops a Retrieval-Augmented Transformer (RAT), aiming to acquire fine-grained feature interactions within and across samples. By retrieving similar samples, we construct augmented input for each target sample. We then build Transformer layers with cascaded attention to capture both intra- and cross-sample feature interactions, facilitating comprehensive reasoning for improved CTR prediction while retaining efficiency. Extensive experiments on real-world datasets substantiate the effectiveness of RAT and suggest its advantage in long-tail scenarios. The code has been open-sourced at \url{https://github.com/YushenLi807/WWW24-RAT}.
Paper Structure (18 sections, 4 equations, 2 figures, 5 tables)

This paper contains 18 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. The Proposed Method
  3. Retrieve Similar Samples as Context
  4. Notes on Avoiding Information Leakage
  5. Construct Retrieval-augmented Input
  6. Intra- and Cross-sample Feature Interaction
  7. Experiments
  8. Experimental Setup
  9. Datasets
  10. Metrics
  11. Baselines
  12. Implementation Details
  13. Comparison with State-of-the-arts
  14. Overall Performance
  15. Performance on Long-tail Data
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Traditional CTR prediction methods only focus on cross-column interaction. In this paper, we focus on a new paradigm of retrieval-augmented (RA) CTR, further incorporating cross-row interaction for more effective prediction.
  • Figure 2: The overview framework of RAT.