Scalability Matters: Overcoming Challenges in InstructGLM with Similarity-Degree-Based Sampling

TL;DR

This work tackles the token-length bottleneck of applying large language models to graph-structured data by eliminating dependence on Graph Neural Networks and introducing SDM-InstructGLM, a GNN-free, instruction-tuned framework. It uses a similarity-degree biased random walk, defined by z_{uv} = cosSim(h_u,h_v) · deg(v) and p(v|u) = softmax(z_{uv}), to sample and encode salient graph regions into token-efficient textual prompts; a structured node ordering with s(v) = cosSim(h_u,h_v) · deg(v) and a hop-aware sampling budget N_sample(h) = min(N_hop(h), T_max / T_avg) further preserves local and global context. Ablation studies show that both cosine similarity and degree weighting are critical, and that SDM-InstructGLM yields substantial gains on sparse graphs like Cora while maintaining robust improvements on PubMed, outperforming the InstructGLM baseline across 1-, 2-, and 3-hop structures. The results demonstrate the feasibility and practicality of LLM-only graph processing for scalable graph reasoning, with code released on GitHub to facilitate adoption and further research.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in various natural language processing tasks; however, their application to graph-related problems remains limited, primarily due to scalability constraints and the absence of dedicated mechanisms for processing graph structures. Existing approaches predominantly integrate LLMs with Graph Neural Networks (GNNs), using GNNs as feature encoders or auxiliary components. However, directly encoding graph structures within LLMs has been underexplored, particularly in the context of large-scale graphs where token limitations hinder effective representation. To address these challenges, we propose SDM-InstructGLM, a novel instruction-tuned Graph Language Model (InstructGLM) framework that enhances scalability and efficiency without relying on GNNs. Our method introduces a similarity-degree-based biased random walk mechanism, which selectively samples and encodes graph information based on node-feature similarity and degree centrality, ensuring an adaptive and structured representation within the LLM. This approach significantly improves token efficiency, mitigates information loss due to random sampling, and enhances performance on graph-based tasks such as node classification and link prediction. Furthermore, our results demonstrate the feasibility of LLM-only graph processing, enabling scalable and interpretable Graph Language Models (GLMs) optimized through instruction-based fine-tuning. This work paves the way for GNN-free approaches to graph learning, leveraging LLMs as standalone graph reasoning models. Our source code is available on GitHub.

Figures (2)

• Figure 1: Illustration of the SDM-InstructGLM framework, which enhances LLM training on graph-structured data using a similarity-degree-based random walk to preserve key structural relationships.
• Figure 2: Illustration of a degree and similarity-biased random walk.