Can LLMs Effectively Leverage Graph Structural Information through Prompts, and Why?

TL;DR

This study interrogates whether LLMs can effectively leverage graph structural information via prompts and why. By constructing a leakage-free arxiv-2023 dataset and comparing with ogbn-arxiv, plus adversarial prompts that linearize or rewire ego-graphs, the authors show data leakage is not the primary driver of LLM gains and that prompts are processed more as linearized paragraphs than as explicit graphs. They further find that structural prompt benefits correlate with local homophily and are most pronounced when the target node has scarce textual features, offering nuanced guidance for prompt design. The work highlights the need for deeper graph-aware encoding or fine-tuning to unlock robust graph-topology understanding in LLMs, with practical implications for graph-based NLP tasks and benchmark design.

Abstract

Large language models (LLMs) are gaining increasing attention for their capability to process graphs with rich text attributes, especially in a zero-shot fashion. Recent studies demonstrate that LLMs obtain decent text classification performance on common text-rich graph benchmarks, and the performance can be improved by appending encoded structural information as natural languages into prompts. We aim to understand why the incorporation of structural information inherent in graph data can improve the prediction performance of LLMs. First, we rule out the concern of data leakage by curating a novel leakage-free dataset and conducting a comparative analysis alongside a previously widely-used dataset. Second, as past work usually encodes the ego-graph by describing the graph structure in natural language, we ask the question: do LLMs understand the graph structure in accordance with the intent of the prompt designers? Third, we investigate why LLMs can improve their performance after incorporating structural information. Our exploration of these questions reveals that (i) there is no substantial evidence that the performance of LLMs is significantly attributed to data leakage; (ii) instead of understanding prompts as graph structures as intended by the prompt designers, LLMs tend to process prompts more as contextual paragraphs and (iii) the most efficient elements of the local neighborhood included in the prompt are phrases that are pertinent to the node label, rather than the graph structure.

Figures (6)

• Figure 1: Proportional distribution of labels in ogbn-arxiv and arxiv-2023 datasets. Each label represents an arXiv Computer Science Category.
• Figure 2: Example of rewiring ego-graphs for node $0$. Three rewiring strategies are evaluated: "random" keeps 1-hop neighbors and randomly connect 2-hop neighbors to 1-hop neighbors; "extreme" keeps 1-hop neighbors and connects all 2-hop neighbors connect to a random 1-hop neighbor; "Path" randomly connects 1-hop neighbors as a path.
• Figure 3: Performance comparison of dropping neighbors using different strategies across arxiv-2023, cora, and ogbn-product datasets. Three dropping strategies are evaluated: "drop same" removes neighbors with the same label as the target node; "drop different" removes neighbors with different labels as the target node; and "drop random" randomly selects neighbors for removal. When percentage is $1$, "drop same" strategy drops all same-label neighbors but preserves all different-label neighbors, and "drop different" strategy drops all different-label neighbors but preserves all same-label neighbors. Details about the strategies are stated in Appendix \ref{['appen:dropping_details']}.
• Figure 4: Performance comparison of dropping neighbors using different strategies on cora dataset. Three dropping strategies are evaluated: (i) 1-hop title+label, (ii) 1-hop title and (iii) 1-hop label
• Figure 5: Performance comparison of dropping neighbors using different strategies on arxiv-2023 dataset. Three dropping strategies are evaluated: (i) 1-hop title+label, (ii) 1-hop title and (iii) 1-hop label