Node Level Graph Autoencoder: Unified Pretraining for Textual Graph Learning

TL;DR

A novel unified unsupervised learning autoencoder framework, named Node Level Graph AutoEncoder (NodeGAE), which employs language models as the backbone of the autoencoder, with pretraining on text reconstruction and adds an auxiliary loss term to make the feature embeddings aware of the local graph structure.

Abstract

Textual graphs are ubiquitous in real-world applications, featuring rich text information with complex relationships, which enables advanced research across various fields. Textual graph representation learning aims to generate low-dimensional feature embeddings from textual graphs that can improve the performance of downstream tasks. A high-quality feature embedding should effectively capture both the structural and the textual information in a textual graph. However, most textual graph dataset benchmarks rely on word2vec techniques to generate feature embeddings, which inherently limits their capabilities. Recent works on textual graph representation learning can be categorized into two folds: supervised and unsupervised methods. Supervised methods finetune a language model on labeled nodes, which have limited capabilities when labeled data is scarce. Unsupervised methods, on the other hand, extract feature embeddings by developing complex training pipelines. To address these limitations, we propose a novel unified unsupervised learning autoencoder framework, named Node Level Graph AutoEncoder (NodeGAE). We employ language models as the backbone of the autoencoder, with pretraining on text reconstruction. Additionally, we add an auxiliary loss term to make the feature embeddings aware of the local graph structure. Our method maintains simplicity in the training process and demonstrates generalizability across diverse textual graphs and downstream tasks. We evaluate our method on two core graph representation learning downstream tasks: node classification and link prediction. Comprehensive experiments demonstrate that our approach substantially enhances the performance of diverse graph neural networks (GNNs) across multiple textual graph datasets.

Figures (6)

• Figure 1: The paradigm for textual graph learning.
• Figure 2: Overview of the autoencoder architecture in NodeGAE. The encoder generates the latent representation $h$ as an extracted feature representation of the input text sequence $\mathcal{T}$. The decoder then reconstructs the original text sequence from the projected $h$. Moreover, the latent representation $h$ is encouraged to learn the structural information of the input text by utilizing the InfoNCE loss function.
• Figure 3: Overview of NodeGAE training pipeline. The training pipeline consists of two stages. In the first stage, the autoencoder is pretrained on the text reconstruction task with the language modeling loss and the InfoNCE loss. In the second stage, a GNN is trained on the feature embeddings extracted from the frozen encoder.
• Figure 4: To compare the convergence rates across different methods, we present the validation accuracy/ROC-AUC, test accuracy/ROC-AUC, and training loss curves for the training process on $h_{shallow}$, $h_{sent-emb}$, and $h_{NodeGAE}$ on the ogbn-arxiv dataset for node classification (NC) and link prediction (LP) tasks. The classifier for NC and LP are RevGAT and MLP respectively.
• Figure 5: The validation ROC-AUC curve for link prediction during the first 25 iterations at the very beginning of the first epoch. We take MLP as the classifier.