Semi-supervised Instruction Tuning for Large Language Models on Text-Attributed Graphs

TL;DR

SIT-Graph tackles label scarcity in text-attributed graph analysis by introducing a semi-supervised instruction tuning pipeline that leverages unlabeled nodes through confidence-filtered pseudo-responses. It is model-agnostic and integrates into existing graph instruction tuning frameworks via an iterative self-training loop that progressively aligns the LLM with graph topology. The method combines graph-aware tokenization with a freezing-based initial IT phase and a robust pseudo-label generation process, yielding consistent improvements (approximately 20–30%) under low-label regimes across multiple baselines and datasets. This approach enables scalable, structure-aware LLM systems for graph tasks, with practical impact for tasks like misinformation detection and community dynamics monitoring in resource-constrained settings.

Abstract

The emergent reasoning capabilities of Large Language Models (LLMs) offer a transformative paradigm for analyzing text-attributed graphs. While instruction tuning is the prevailing method for adapting pre-trained LLMs to graph learning tasks like node classification, it requires a substantial volume of annotated (INSTRUCTION, OUTPUT) pairs deriving from labeled nodes. This requirement is particularly prohibitive in the social domain, where obtaining expert labels for sensitive or evolving content is costly and slow. Furthermore, standard graph instruction tuning fails to exploit the vast amount of unlabeled nodes, which contain latent correlations due to edge connections that are beneficial for downstream predictions. To bridge this gap, we propose a novel Semi-supervised Instruction Tuning pipeline for Graph Learning, named SIT-Graph. Notably, SIT-Graph is model-agnostic and can be seamlessly integrated into any graph instruction tuning method that utilizes LLMs as the predictor. SIT-Graph operates via an iterative self-training process. Initially, the model is fine-tuned using instruction pairs constructed solely from the labeled nodes. Then it generates confidence-filtered pseudo-responses for unlabeled nodes to strategically augment the dataset for the next round of fine-tuning. Finally, this iterative refinement progressively aligns the LLM with the underlying node correlations. Extensive experiments demonstrate that when incorporated into state-of-the-art graph instruction tuning methods, SIT-Graph significantly enhances their performance on text-attributed graph benchmarks, achieving over 20% improvement under the low label ratio settings.

Figures (3)

• Figure 1: Comparison of existing IT methods (a) and ours (b).
• Figure 2: Semi-supervised instruction tuning pipeline for graph learning (SIT-Graph). First, it conducts instruction tuning, adapting a pre-trained LLM to map combined graph tokens and text tokens to the handcrafted response for labeled nodes (Sec. \ref{['sec:it']}). Second, the tuned model generates pseudo-responses and confidence scores for unlabeled nodes (Sec. \ref{['sec:pseudo_gen']}). Finally, high-quality pseudo-responses are added to the augmented dataset, and the model evolves in a self-training manner (Sec. \ref{['sec:self-training']}).
• Figure 3: Impact of the labeled-to-unlabeled node ratio on accuracy improvement.