Hgformer: Hyperbolic Graph Transformer for Recommendation

TL;DR

Hgformer addresses cold-start and long-tail challenges in recommender systems by embedding users and items in a hyperbolic manifold and coupling local and global information through a Light Hyperbolic Graph Convolutional Network (LHGCN) and a hyperbolic cross-attention transformer. It introduces a linear-time, unbiased approximation of hyperbolic attention via random features, with theoretical guarantees, achieving complexity $O(M+N)$ while preserving performance. Empirical results on six datasets show consistent improvements over traditional, Euclidean GNNs, hyperbolic GNNs, and graph transformers, with notable gains on tail items and cross-domain compatibility. This work demonstrates the effectiveness of hyperbolic geometry for modeling hierarchical, power-law data in recommendation and points to future directions in transfer learning and hybrid transformer integration.

Abstract

The cold start problem is a challenging problem faced by most modern recommender systems. By leveraging knowledge from other domains, cross-domain recommendation can be an effective method to alleviate the cold start problem. However, the modelling distortion for long-tail data, which is widely present in recommender systems, is often overlooked in cross-domain recommendation. In this research, we propose a hyperbolic manifold based cross-domain collaborative filtering model using BiTGCF as the base model. We introduce the hyperbolic manifold and construct new propagation layer and transfer layer to address these challenges. The significant performance improvements across various datasets compared to the baseline models demonstrate the effectiveness of our proposed model.

Figures (7)

• Figure 1: This figure highlights two key challenges that traditional GNNs often face when dealing with long-tail items. The first one is the message-passing paradigm in local neighborhoods, which prevents the model from transferring information from less popular items to most users, leading to poor recommendations for these items. The second one is that the embeddings in Euclidean space may fail to capture the long distance in graphs, and will lead to serious information distortion in graph structure.
• Figure 2: Figure(a) shows the overall architecture of Hgformer. We first map the user and item embeddings into a hyperbolic manifold. Then, we use several layers of LHGCN to capture the local information of the interaction graph and use a hyperbolic transformer to capture the overall information of the interaction graph. Finally, we combine these two information and make predictions on the hyperbolic manifold. Figure (b) shows the difference between HGCF and LHGCN in graph convolution. In HGCF, embeddings are mapped from the hyperbolic manifold to the North Pole point’s tangent space for aggregation, then back to the hyperbolic manifold, causing information distortion due to the local approximation. LHGCN, on the other hand, aggregates information directly in the hyperbolic manifold. Figure (c) shows the process of hyperbolic cross-attention. Unlike GNN-based models that only consider items a user has interacted with, Hyperbolic Cross-Attention takes both the interacted and non-interacted items into account.
• Figure 3: This figure shows how we reduce the complexity of hyperbolic graph transformers to linear. We first use $\phi$ to transform the similarity matrix into the multiplication of two smaller matrices and then, we change the order of computation order to reduce the computation complexity.
• Figure 4: To validate the contribution of each module in the model, we individually removed the LHGCN and the Hyperbolic Transformer for evaluation. In the figure, 'w/o LHGCN' and 'w/o Transformer' represent the effects after removing the LHGCN and the Hyperbolic Transformer, respectively. Furthermore, to validate the effectiveness of LHGCN, we also compared LHGCN with HGCF. Specifically, we replaced LHGCN in Hgformer with HGCF and tested the performance and in the figure, it is noted as 'HGCF+Transformer'.
• Figure 5: We define the top 20% of items in terms of popularity as head items and the rest as tail items. We calculated the proportion of head items and tail items among all recommended items for each model. To further analyze the performance of Hgformer on tail items, we evaluate the performance of three hyperbolic-based models on tail items.