Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs

Jianxiang Yu, Yuxiang Ren, Chenghua Gong, Jiaqi Tan, Xiang Li, Xuecang Zhang

TL;DR

This work tackles few-shot node classification on text-attributed graphs by introducing LLM4NG, a lightweight, plug-and-play paradigm that uses Large Language Models to generate labeled samples from class descriptions and a simple edge predictor to connect these samples to the original graph. The augmented graph, together with sentence-embedded representations, enables effective supervision signal propagation through any Graph Neural Network, even with extremely limited labels. Empirical results on Cora, PubMed, and ogbn-arxiv demonstrate substantial gains in zero- and few-shot regimes, including notable improvements over strong baselines and significant reductions in LLM-invocation costs relative to prior LLM-based augmentation methods. The approach offers a practical pathway to leverage rich external knowledge encoded in LLMs for graph-based learning without heavy computational burdens, with potential extensions to handle imbalanced data and labeling noise.

Abstract

Text-attributed graphs have recently garnered significant attention due to their wide range of applications in web domains. Existing methodologies employ word embedding models for acquiring text representations as node features, which are subsequently fed into Graph Neural Networks (GNNs) for training. Recently, the advent of Large Language Models (LLMs) has introduced their powerful capabilities in information retrieval and text generation, which can greatly enhance the text attributes of graph data. Furthermore, the acquisition and labeling of extensive datasets are both costly and time-consuming endeavors. Consequently, few-shot learning has emerged as a crucial problem in the context of graph learning tasks. In order to tackle this challenge, we propose a lightweight paradigm called LLM4NG, which adopts a plug-and-play approach to empower text-attributed graphs through node generation using LLMs. Specifically, we utilize LLMs to extract semantic information from the labels and generate samples that belong to these categories as exemplars. Subsequently, we employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph. This approach harnesses LLMs for enhancing class-level information and seamlessly introduces labeled nodes and edges without modifying the raw dataset, thereby facilitating the node classification task in few-shot scenarios. Extensive experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios. For instance, in the 1-shot setting of the ogbn-arxiv dataset, LLM4NG achieves a 76% improvement over the baseline model.

Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs

TL;DR

This work tackles few-shot node classification on text-attributed graphs by introducing LLM4NG, a lightweight, plug-and-play paradigm that uses Large Language Models to generate labeled samples from class descriptions and a simple edge predictor to connect these samples to the original graph. The augmented graph, together with sentence-embedded representations, enables effective supervision signal propagation through any Graph Neural Network, even with extremely limited labels. Empirical results on Cora, PubMed, and ogbn-arxiv demonstrate substantial gains in zero- and few-shot regimes, including notable improvements over strong baselines and significant reductions in LLM-invocation costs relative to prior LLM-based augmentation methods. The approach offers a practical pathway to leverage rich external knowledge encoded in LLMs for graph-based learning without heavy computational burdens, with potential extensions to handle imbalanced data and labeling noise.

Abstract

Text-attributed graphs have recently garnered significant attention due to their wide range of applications in web domains. Existing methodologies employ word embedding models for acquiring text representations as node features, which are subsequently fed into Graph Neural Networks (GNNs) for training. Recently, the advent of Large Language Models (LLMs) has introduced their powerful capabilities in information retrieval and text generation, which can greatly enhance the text attributes of graph data. Furthermore, the acquisition and labeling of extensive datasets are both costly and time-consuming endeavors. Consequently, few-shot learning has emerged as a crucial problem in the context of graph learning tasks. In order to tackle this challenge, we propose a lightweight paradigm called LLM4NG, which adopts a plug-and-play approach to empower text-attributed graphs through node generation using LLMs. Specifically, we utilize LLMs to extract semantic information from the labels and generate samples that belong to these categories as exemplars. Subsequently, we employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph. This approach harnesses LLMs for enhancing class-level information and seamlessly introduces labeled nodes and edges without modifying the raw dataset, thereby facilitating the node classification task in few-shot scenarios. Extensive experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios. For instance, in the 1-shot setting of the ogbn-arxiv dataset, LLM4NG achieves a 76% improvement over the baseline model.
Paper Structure (28 sections, 7 equations, 3 figures, 9 tables)

This paper contains 28 sections, 7 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Models on TAGs.
  4. Few-shot learning on Graphs.
  5. Preliminary
  6. Text-Attributed Graphs.
  7. Few-shot Node Classification.
  8. Graph Neural Network Architecture
  9. The Proposed Paradigm
  10. Sample Generation from LLM
  11. Node representation initialization with LM
  12. Edge Predictor
  13. LLM4NG
  14. Experiments
  15. Datasets and Baselines
  16. ...and 13 more sections

Figures (3)

  • Figure 1: The overall paradigm of LLM4NG.
  • Figure 2: Impact of (a) the quantity of generated samples and (b) LLMs’ stochasticity on model performance.
  • Figure 3: Hyper-parameter sensitivity analysis on the Pubmed dataset. $\delta$ is the similarity threshold and $k$ is the number of connected edges.