Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Graph Neural Networks with Limited Labeled Data by Actively Distilling Knowledge from Large Language Models

Quan Li, Tianxiang Zhao, Lingwei Chen, Junjie Xu, Suhang Wang

TL;DR

This work tackles few-shot node classification on graphs by integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs). It treats LLMs as teachers to provide soft labels, logits, and rationales, which are distilled into GNNs, and it introduces a Graph-LLM-based active learning scheme to selectively query LLMs for high-impact nodes under a budget. The method combines a supervised loss with knowledge-distillation and feature-alignment losses to leverage unlabeled data and LLM explanations, achieving state-of-the-art results on Cora, Citeseer, and PubMed under 1–7 shot settings. The approach demonstrates that zero-shot reasoning and structured rationales from LLMs can significantly boost GNN performance when labeled data are scarce, with practical benefits for text-attributed graphs and label-efficient learning.

Abstract

Graphs are pervasive in the real-world, such as social network analysis, bioinformatics, and knowledge graphs. Graph neural networks (GNNs) have great ability in node classification, a fundamental task on graphs. Unfortunately, conventional GNNs still face challenges in scenarios with few labeled nodes, despite the prevalence of few-shot node classification tasks in real-world applications. To address this challenge, various approaches have been proposed, including graph meta-learning, transfer learning, and methods based on Large Language Models (LLMs). However, traditional meta-learning and transfer learning methods often require prior knowledge from base classes or fail to exploit the potential advantages of unlabeled nodes. Meanwhile, LLM-based methods may overlook the zero-shot capabilities of LLMs and rely heavily on the quality of generated contexts. In this paper, we propose a novel approach that integrates LLMs and GNNs, leveraging the zero-shot inference and reasoning capabilities of LLMs and employing a Graph-LLM-based active learning paradigm to enhance GNNs' performance. Extensive experiments demonstrate the effectiveness of our model in improving node classification accuracy with considerably limited labeled data, surpassing state-of-the-art baselines by significant margins.

Enhancing Graph Neural Networks with Limited Labeled Data by Actively Distilling Knowledge from Large Language Models

TL;DR

This work tackles few-shot node classification on graphs by integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs). It treats LLMs as teachers to provide soft labels, logits, and rationales, which are distilled into GNNs, and it introduces a Graph-LLM-based active learning scheme to selectively query LLMs for high-impact nodes under a budget. The method combines a supervised loss with knowledge-distillation and feature-alignment losses to leverage unlabeled data and LLM explanations, achieving state-of-the-art results on Cora, Citeseer, and PubMed under 1–7 shot settings. The approach demonstrates that zero-shot reasoning and structured rationales from LLMs can significantly boost GNN performance when labeled data are scarce, with practical benefits for text-attributed graphs and label-efficient learning.

Abstract

Graphs are pervasive in the real-world, such as social network analysis, bioinformatics, and knowledge graphs. Graph neural networks (GNNs) have great ability in node classification, a fundamental task on graphs. Unfortunately, conventional GNNs still face challenges in scenarios with few labeled nodes, despite the prevalence of few-shot node classification tasks in real-world applications. To address this challenge, various approaches have been proposed, including graph meta-learning, transfer learning, and methods based on Large Language Models (LLMs). However, traditional meta-learning and transfer learning methods often require prior knowledge from base classes or fail to exploit the potential advantages of unlabeled nodes. Meanwhile, LLM-based methods may overlook the zero-shot capabilities of LLMs and rely heavily on the quality of generated contexts. In this paper, we propose a novel approach that integrates LLMs and GNNs, leveraging the zero-shot inference and reasoning capabilities of LLMs and employing a Graph-LLM-based active learning paradigm to enhance GNNs' performance. Extensive experiments demonstrate the effectiveness of our model in improving node classification accuracy with considerably limited labeled data, surpassing state-of-the-art baselines by significant margins.
Paper Structure (24 sections, 14 equations, 3 figures, 7 tables)

This paper contains 24 sections, 14 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Understanding LLM's Capability
  5. Problem Statement
  6. Proposed Model
  7. Base GNN Classifier
  8. Obtaining Knowledge from LLM
  9. Soft Labels and Logits Generation
  10. Rationales for Feature Enhancement
  11. Distilling Knowledge to GNN
  12. Loss for Knowledge Distillation
  13. Loss for Feature Alignment
  14. Objective Function of Proposed Framework
  15. Graph-LLM-based Active learning
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Preliminary experiments: the impact of the degrees (left) and homophily ratio (right) to LLMs
  • Figure 2: An illustration of the proposed framework
  • Figure 3: Hyper-parameters evaluation: (a) Alpha (b) Beta (c) Temperature (d) Budget Size