Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scale-Free Graph-Language Models

Jianglin Lu, Yixuan Liu, Yitian Zhang, Yun Fu

TL;DR

This paper tackles graph-based semi-supervised learning with GLMs by introducing Scale-Free Graph-Language (SFGL), which unifies graph generation and text embedding around a scale-free prior. It shows that a cosine-based $k$-NN graph can approximate scale-free structure (power-law degree distribution $P(\theta) \propto \theta^{-\alpha}$ with $\alpha$ around $2.7$–$3.3$) and couples this with a graph-based pseudo-labeler to supervise LM fine-tuning, further enhanced by GPT-3.5 augmentations in an iterative loop. Extensive experiments on four citation datasets demonstrate that SFGL and its GPT-augmented variant achieve state-of-the-art performance, particularly under low supervision, and that the cosine-based KNN graphs effectively capture the necessary structure. The work highlights a synergistic integration of GNNs and LMs under a real, simple structural prior and provides insights into when artificial graphs may be advantageous for LM fine-tuning.

Abstract

Graph-language models (GLMs) have demonstrated great potential in graph-based semi-supervised learning. A typical GLM consists of two key stages: graph generation and text embedding, which are usually implemented by inferring a latent graph and finetuning a language model (LM), respectively. However, the former often relies on artificial assumptions about the underlying edge distribution, while the latter requires extensive data annotations. To tackle these challenges, this paper introduces a novel GLM that integrates graph generation and text embedding within a unified framework. Specifically, for graph generation, we leverage an inherent characteristic of real edge distribution--the scale-free property--as a structural prior. We unexpectedly find that this natural property can be effectively approximated by a simple k-nearest neighbor (KNN) graph. For text embedding, we develop a graph-based pseudo-labeler that utilizes scale-free graphs to provide complementary supervision for improved LM finetuning. Extensive experiments on representative datasets validate our findings on the scale-free structural approximation of KNN graphs and demonstrate the effectiveness of integrating graph generation and text embedding with a real structural prior. Our code is available at https://github.com/Jianglin954/SFGL.

Scale-Free Graph-Language Models

TL;DR

This paper tackles graph-based semi-supervised learning with GLMs by introducing Scale-Free Graph-Language (SFGL), which unifies graph generation and text embedding around a scale-free prior. It shows that a cosine-based -NN graph can approximate scale-free structure (power-law degree distribution with around –) and couples this with a graph-based pseudo-labeler to supervise LM fine-tuning, further enhanced by GPT-3.5 augmentations in an iterative loop. Extensive experiments on four citation datasets demonstrate that SFGL and its GPT-augmented variant achieve state-of-the-art performance, particularly under low supervision, and that the cosine-based KNN graphs effectively capture the necessary structure. The work highlights a synergistic integration of GNNs and LMs under a real, simple structural prior and provides insights into when artificial graphs may be advantageous for LM fine-tuning.

Abstract

Graph-language models (GLMs) have demonstrated great potential in graph-based semi-supervised learning. A typical GLM consists of two key stages: graph generation and text embedding, which are usually implemented by inferring a latent graph and finetuning a language model (LM), respectively. However, the former often relies on artificial assumptions about the underlying edge distribution, while the latter requires extensive data annotations. To tackle these challenges, this paper introduces a novel GLM that integrates graph generation and text embedding within a unified framework. Specifically, for graph generation, we leverage an inherent characteristic of real edge distribution--the scale-free property--as a structural prior. We unexpectedly find that this natural property can be effectively approximated by a simple k-nearest neighbor (KNN) graph. For text embedding, we develop a graph-based pseudo-labeler that utilizes scale-free graphs to provide complementary supervision for improved LM finetuning. Extensive experiments on representative datasets validate our findings on the scale-free structural approximation of KNN graphs and demonstrate the effectiveness of integrating graph generation and text embedding with a real structural prior. Our code is available at https://github.com/Jianglin954/SFGL.

Paper Structure

This paper contains 22 sections, 4 theorems, 3 equations, 8 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Definition
  4. Scale-Free Networks
  5. K-Nearest Neighbor Graphs
  6. Scale-Free Graph-Language Model
  7. Scale-Free Graph Generation
  8. Improved Text Embedding
  9. Classification
  10. Experiments
  11. Experimental Settings
  12. Comparison Analysis
  13. Ablation Study
  14. Related Work
  15. Conclusion
  16. ...and 7 more sections

Key Result

Proposition 1

For a scale-free network, assuming that there is at most one node $v$ whose degree belongs to $[\theta_{max}, \infty)$, we have $\theta_{max} = \theta_{min} |\mathcal{V}|^{\frac{1}{\alpha-1}}$, where $\theta_{max}$ and $\theta_{min}$ refer to the maximum and minimal degree in the scale-free network.

Figures (8)

  • Figure 1: Illustration of a scale-free network, highlighting a few hubs and a large number of small nodes.
  • Figure 2: Uniform node distribution within structured arrangements causes consistent connectivity.
  • Figure 3: Curve fitting to the in-degree distribution of a KNN graph ($k=5$) using Euclidean distance as the metric.
  • Figure 4: In-degree distribution of KNN graphs on different datasets, where the top-$5$ largest hubs are marked in red.
  • Figure 5: Curve fitting to the in-degree distribution of KNN graphs on different datasets.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Proposition 1
  • Proposition 2
  • Proposition 1
  • proof
  • Proposition 2
  • proof