Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rankformer: A Graph Transformer for Recommendation based on Ranking Objective

Sirui Chen, Shen Han, Jiawei Chen, Binbin Hu, Sheng Zhou, Gang Wang, Yan Feng, Chun Chen, Can Wang

TL;DR

Rankformer addresses the gap between recommender architectures and the core ranking objective by deriving a gradient-inspired graph transformer layer that updates user and item embeddings toward better ranking. The model incorporates positive and negative interaction signals with specialized attention weights and reveals a theoretical link to Levenberg–Marquardt optimization; an acceleration scheme reduces complexity to linear in the number of positive instances. Empirical results on four real-world datasets show Rankformer outperforms state-of-the-art baselines, with Rankformer-CL offering additional gains via contrastive learning. The work provides a principled mechanism to embed ranking signals into architecture design and suggests promising extensions to sequential and LLM-based recommendation systems.

Abstract

Recommender Systems (RS) aim to generate personalized ranked lists for each user and are evaluated using ranking metrics. Although personalized ranking is a fundamental aspect of RS, this critical property is often overlooked in the design of model architectures. To address this issue, we propose Rankformer, a ranking-inspired recommendation model. The architecture of Rankformer is inspired by the gradient of the ranking objective, embodying a unique (graph) transformer architecture -- it leverages global information from all users and items to produce more informative representations and employs specific attention weights to guide the evolution of embeddings towards improved ranking performance. We further develop an acceleration algorithm for Rankformer, reducing its complexity to a linear level with respect to the number of positive instances. Extensive experimental results demonstrate that Rankformer outperforms state-of-the-art methods. The code is available at https://github.com/StupidThree/Rankformer.

Rankformer: A Graph Transformer for Recommendation based on Ranking Objective

TL;DR

Rankformer addresses the gap between recommender architectures and the core ranking objective by deriving a gradient-inspired graph transformer layer that updates user and item embeddings toward better ranking. The model incorporates positive and negative interaction signals with specialized attention weights and reveals a theoretical link to Levenberg–Marquardt optimization; an acceleration scheme reduces complexity to linear in the number of positive instances. Empirical results on four real-world datasets show Rankformer outperforms state-of-the-art baselines, with Rankformer-CL offering additional gains via contrastive learning. The work provides a principled mechanism to embed ranking signals into architecture design and suggests promising extensions to sequential and LLM-based recommendation systems.

Abstract

Recommender Systems (RS) aim to generate personalized ranked lists for each user and are evaluated using ranking metrics. Although personalized ranking is a fundamental aspect of RS, this critical property is often overlooked in the design of model architectures. To address this issue, we propose Rankformer, a ranking-inspired recommendation model. The architecture of Rankformer is inspired by the gradient of the ranking objective, embodying a unique (graph) transformer architecture -- it leverages global information from all users and items to produce more informative representations and employs specific attention weights to guide the evolution of embeddings towards improved ranking performance. We further develop an acceleration algorithm for Rankformer, reducing its complexity to a linear level with respect to the number of positive instances. Extensive experimental results demonstrate that Rankformer outperforms state-of-the-art methods. The code is available at https://github.com/StupidThree/Rankformer.

Paper Structure

This paper contains 26 sections, 1 theorem, 19 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Background on Recommender Systems
  4. Transformer Architecture
  5. Methodology
  6. Rankformer Layer
  7. Disscussion
  8. Fast Implementation
  9. Experiments
  10. Experimental Settings
  11. Datasets
  12. Metrics
  13. Baselines
  14. Performance Comparison (RQ1)
  15. Ablation Study (RQ2)
  16. ...and 11 more sections

Key Result

lemma 1

Performing gradient descent on $\mathcal{L}_{R}(\theta)$ is equivalent to using the Levenberg-Marquardt algorithm to perform gradient descent on $\mathcal{L}(\theta)$.

Figures (3)

  • Figure 1: Performance in terms of $NDCG@20$ when using Rankformer and LightGCN with different numbers of layers for randomly initialized representations without training.
  • Figure 2: Performance of Rankformer in terms of $NDCG@20$ with different layers $L$ and hyperparameter $\tau$.
  • Figure 3: Performance in terms of $NDCG@20$ with different hyperparameter $\alpha$.

Theorems & Definitions (1)

  • lemma 1