Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Evaluation Perspective on GNNs-based Recommender Systems through the Topology of the User-Item Graph

Daniele Malitesta, Claudio Pomo, Vito Walter Anelli, Alberto Carlo Maria Mancino, Tommaso Di Noia, Eugenio Di Sciascio

TL;DR

This work introduces a topology-centered evaluation framework for GNN-based recommender systems, investigating how the topology of the user-item graph shapes performance. By constructing 1,800 reduced datasets via node- and edge-dropout from Yelp2018, Gowalla, and Amazon-Book, and evaluating three GNNs (LightGCN, DGCF, SVD-GCN), the authors link classical and topological data characteristics to Recall@20 using a linear explanatory model. The study finds strong, statistically significant associations, notably that average degree and certain topological cues (clustering, assortativity) influence performance, often with the factorization component playing a key role. The work provides a reproducible pipeline and code, offering a new lens for understanding when and why topology-aware GNN recommendations succeed and guiding future enhancements in topology-informed recommender design.

Abstract

Recently, graph neural networks (GNNs)-based recommender systems have encountered great success in recommendation. As the number of GNNs approaches rises, some works have started questioning the theoretical and empirical reasons behind their superior performance. Nevertheless, this investigation still disregards that GNNs treat the recommendation data as a topological graph structure. Building on this assumption, in this work, we provide a novel evaluation perspective on GNNs-based recommendation, which investigates the impact of the graph topology on the recommendation performance. To this end, we select some (topological) properties of the recommendation data and three GNNs-based recommender systems (i.e., LightGCN, DGCF, and SVD-GCN). Then, starting from three popular recommendation datasets (i.e., Yelp2018, Gowalla, and Amazon-Book) we sample them to obtain 1,800 size-reduced datasets that still resemble the original ones but can encompass a wider range of topological structures. We use this procedure to build a large pool of samples for which data characteristics and recommendation performance of the selected GNNs models are measured. Through an explanatory framework, we find strong correspondences between graph topology and GNNs performance, offering a novel evaluation perspective on these models.

A Novel Evaluation Perspective on GNNs-based Recommender Systems through the Topology of the User-Item Graph

TL;DR

This work introduces a topology-centered evaluation framework for GNN-based recommender systems, investigating how the topology of the user-item graph shapes performance. By constructing 1,800 reduced datasets via node- and edge-dropout from Yelp2018, Gowalla, and Amazon-Book, and evaluating three GNNs (LightGCN, DGCF, SVD-GCN), the authors link classical and topological data characteristics to Recall@20 using a linear explanatory model. The study finds strong, statistically significant associations, notably that average degree and certain topological cues (clustering, assortativity) influence performance, often with the factorization component playing a key role. The work provides a reproducible pipeline and code, offering a new lens for understanding when and why topology-aware GNN recommendations succeed and guiding future enhancements in topology-informed recommender design.

Abstract

Recently, graph neural networks (GNNs)-based recommender systems have encountered great success in recommendation. As the number of GNNs approaches rises, some works have started questioning the theoretical and empirical reasons behind their superior performance. Nevertheless, this investigation still disregards that GNNs treat the recommendation data as a topological graph structure. Building on this assumption, in this work, we provide a novel evaluation perspective on GNNs-based recommendation, which investigates the impact of the graph topology on the recommendation performance. To this end, we select some (topological) properties of the recommendation data and three GNNs-based recommender systems (i.e., LightGCN, DGCF, and SVD-GCN). Then, starting from three popular recommendation datasets (i.e., Yelp2018, Gowalla, and Amazon-Book) we sample them to obtain 1,800 size-reduced datasets that still resemble the original ones but can encompass a wider range of topological structures. We use this procedure to build a large pool of samples for which data characteristics and recommendation performance of the selected GNNs models are measured. Through an explanatory framework, we find strong correspondences between graph topology and GNNs performance, offering a novel evaluation perspective on these models.
Paper Structure (17 sections, 15 equations, 3 figures, 1 table)

This paper contains 17 sections, 15 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction and Motivations
  2. Our contributions
  3. Topological data characteristics
  4. Topological Characteristics in Graph-based recommendation
  5. Proposed Evaluation Pipeline
  6. Step 1: Recommendation data collection
  7. Step 2: Characteristics calculation
  8. Step 3: Train/test GNNs-based models
  9. Step 4: Explanatory model
  10. Results and Discussion
  11. Impact of characteristics (RQ1)
  12. Classical characteristics
  13. Topological characteristics
  14. Influence of node- and edge-dropout (RQ2)
  15. Theoretical intuition
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Pearson correlation of the selected characteristics. Many values in $[-0.5, 0.5]$ indicate loosely correlated pairs.
  • Figure 2: Visual representation of the impact of dataset characteristics on the recommendation performance (Recall@20) of GNNs-based recommender systems, for each dataset/model setting. Bar plot length and direction represent the impact magnitude and whether there is a direct/inverse correspondence between characteristic and performance. Finally, the darker the bar plots, the higher their statistical significance.
  • Figure 3: Node degree probability distribution on Gowalla. The black points (i.e., the real data) would be approximated by a function in-between the power-law and the exponential.