Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?

TL;DR

This work interrogates whether graph self-supervised learning (SSL) adheres to neural scaling laws, a cornerstone for building Graph Foundation Models. By benchmarking representative Graph SSL methods under data- and model-scaling regimes on graph classification, it finds that downstream performance does not scale with data or parameter count, even as SSL losses continue to improve. In contrast, SSL loss shows clearer, method- and architecture-dependent scaling with data and with model depth/aggregation, highlighting a gap between pretraining objectives and downstream tasks. The study suggests that future Graph Foundation Model design should prioritize SSL task design and backbone architecture (e.g., graph transformers) to align SSL progress with downstream gains.

Abstract

Self-supervised learning~(SSL) is essential to obtain foundation models in NLP and CV domains via effectively leveraging knowledge in large-scale unlabeled data. The reason for its success is that a suitable SSL design can help the model to follow the neural scaling law, i.e., the performance consistently improves with increasing model and dataset sizes. However, it remains a mystery whether existing SSL in the graph domain can follow the scaling behavior toward building Graph Foundation Models~(GFMs) with large-scale pre-training. In this study, we examine whether existing graph SSL techniques can follow the neural scaling behavior with the potential to serve as the essential component for GFMs. Our benchmark includes comprehensive SSL technique implementations with analysis conducted on both the conventional SSL setting and many new settings adopted in other domains. Surprisingly, despite the SSL loss continuously decreasing, no existing graph SSL techniques follow the neural scaling behavior on the downstream performance. The model performance only merely fluctuates on different data scales and model scales. Instead of the scales, the key factors influencing the performance are the choices of model architecture and pretext task design. This paper examines existing SSL techniques for the feasibility of Graph SSL techniques in developing GFMs and opens a new direction for graph SSL design with the new evaluation prototype. Our code implementation is available online to ease reproducibility on https://github.com/GraphSSLScaling/GraphSSLScaling.

Figures (29)

• Figure 1: Data Scaling of Performance on ogbg-molhiv.x-axis indicates the data amount used for pre-training and y-axis indicates the downstream performance. No obvious scaling behavior can be observed.
• Figure 2: Data Scaling of Performance on reddit-threads. x-axis indicates the data amount used for pre-training and y-axis indicates the downstream performance. No obvious scaling behavior can be observed.
• Figure 3: Data Scaling of SSL Loss on ogbg-molhiv. x-axis indicates the data amount used for pre-training and y-axis indicates the SSL Loss on the held-out test data. More obvious scaling behavior can be observed compared to performance.
• Figure 4: Data Scaling of SSL Loss on reddit-threads.x-axis indicates the data amount used for pre-training and y-axis indicates the SSL Loss on the held-out test data. More obvious scaling behavior can be observed compared to performance.
• Figure 5: Performance on reddit-threads. x-axis denotes the total number of parameters and y-axis denotes the downstream performance. Obvious scaling behavior can be observed on all methods except GraphMAE. The $R^2$ values for each method are listed as follows, GraphCL:0.99,JOAO:0.99,InfoGraph:0.95,GraphMAE:0.36