InfoNCE is a Free Lunch for Semantically guided Graph Contrastive Learning

TL;DR

This work addresses sampling bias in graph contrastive learning arising from augmentation-based positive/negative sampling by framing GCL as Positive-Unlabeled learning. It introduces IFL-GCL, which uses InfoNCE as a 'free-lunch' to extract semantic information via a density-ratio view $r(oldsymbol{x})$ under the Invariance of Order, enabling semantically guided resampling of $D_U^+$ and a corrected objective $L^{corrected}$. The approach yields strong improvements in both standard graph pre-training and LLM-enhanced graph frameworks, with gains up to $9.05\%$ on IID and OOD tasks, and demonstrates resilience across diverse datasets and model scales. This semantic-guided bias correction has practical implications for more transferable graph representations and graph foundation models that leverage LLMs as enhancers.

Abstract

As an important graph pre-training method, Graph Contrastive Learning (GCL) continues to play a crucial role in the ongoing surge of research on graph foundation models or LLM as enhancer for graphs. Traditional GCL optimizes InfoNCE by using augmentations to define self-supervised tasks, treating augmented pairs as positive samples and others as negative. However, this leads to semantically similar pairs being classified as negative, causing significant sampling bias and limiting performance. In this paper, we argue that GCL is essentially a Positive-Unlabeled (PU) learning problem, where the definition of self-supervised tasks should be semantically guided, i.e., augmented samples with similar semantics are considered positive, while others, with unknown semantics, are treated as unlabeled. From this perspective, the key lies in how to extract semantic information. To achieve this, we propose IFL-GCL, using InfoNCE as a "free lunch" to extract semantic information. Specifically, We first prove that under InfoNCE, the representation similarity of node pairs aligns with the probability that the corresponding contrastive sample is positive. Then we redefine the maximum likelihood objective based on the corrected samples, leading to a new InfoNCE loss function. Extensive experiments on both the graph pretraining framework and LLM as an enhancer show significantly improvements of IFL-GCL in both IID and OOD scenarios, achieving up to a 9.05% improvement, validating the effectiveness of semantically guided. Code for IFL-GCL is publicly available at: https://github.com/Camel-Prince/IFL-GCL.

Figures (5)

• Figure 1: (a) illustrates the sampling bias via a case example; (b) validates sampling bias by comparing the nodes' representation similarity between augmented samples and top-20 non-augmented ones by a supervised graph encoder on GOODTwitch; (c) shows the harmfulness of sampling bias by comparing downstream task performances of traditional GCL baselines and our bias-corrected methods.
• Figure 2: Semantics similarity matrix of $\mathbf{G}^{aug_1}$ and $\mathbf{G}^{aug_2}$ after supervised-learning which is rearranged with the $D^{aug+}$ at the first column and decently sorted $D^{aug-}$ as follows.
• Figure 3: Analysis of semantic similarity of $D_U^+$ during training. The gray dashed line indicates the training epoch of optimal checkpoint.
• Figure 4: Analysis of Hyper-parameters: number of warm-up epochs $M$ and number of update-interval epochs $K$.
• Figure 5: Analysis of Hyper-parameter: threshold $t_s$