<SOG_k>: One LLM Token for Explicit Graph Structural Understanding

TL;DR

This paper introduces <SO_Gk>, a single, topology-aware token that encapsulates graph structure within the same token space as natural language, addressing token inefficiency and structural misalignment in LLM-based graph understanding. A topology-aware tokenizer maps graph topology to a discrete codebook token, and hybrid structure QA corpora align these tokens with text representations, enabling explicit, interpretable topology processing. Through extensive experiments on MoleculeNet benchmarks, the approach yields substantial gains ($9.9\%-41.4\%$) over baselines, while maintaining interpretability and extending effectively to node-level tasks. The method offers a scalable, cross-domain Graph-to-Token framework that bridges graph structure and language models for reliable reasoning about structured data.

Abstract

Large language models show great potential in unstructured data understanding, but still face significant challenges with graphs due to their structural hallucination. Existing approaches mainly either verbalize graphs into natural language, which leads to excessive token consumption and scattered attention, or transform graphs into trainable continuous embeddings (i.e., soft prompt), but exhibit severe misalignment with original text tokens. To solve this problem, we propose to incorporate one special token <SOG_k> to fully represent the Structure Of Graph within a unified token space, facilitating explicit topology input and structural information sharing. Specifically, we propose a topology-aware structural tokenizer that maps each graph topology into a highly selective single token. Afterwards, we construct a set of hybrid structure Question-Answering corpora to align new structural tokens with existing text tokens. With this approach, <SOG_k> empowers LLMs to understand, generate, and reason in a concise and accurate manner. Extensive experiments on five graph-level benchmarks demonstrate the superiority of our method, achieving a performance improvement of 9.9% to 41.4% compared to the baselines while exhibiting interpretability and consistency. Furthermore, our method provides a flexible extension to node-level tasks, enabling both global and local structural understanding. The codebase is publicly available at https://github.com/Jingyao-Wu/SOG.

Figures (11)

• Figure 1: The overall architecture for LLM understanding with structural token <SO$\mathcal{G}_k$>.
• Figure 2: A toy example of performance variation across the selection of different structural tokens. More examples are shown in Figure \ref{['fig:ablation_cases1']}, \ref{['fig:ablation_cases2']}, \ref{['fig:ablation_cases3']} in Appendix \ref{['app:Additional Case Studies']}.
• Figure 3: Ablation of tuning the LLM with different hybrid QAs corpora on five datasets.
• Figure 4: Correlation heatmap of first 50 structural tokens.
• Figure 5: Molecules sharing the same scaffold consistently select the same structural token. More examples are shown in \ref{['fig:molecules_train_cases2']}, \ref{['fig:molecules_train_cases1']} in Appendix \ref{['app:Additional Case Studies']}.