A Hierarchical Language Model For Interpretable Graph Reasoning

TL;DR

Comprehensive evaluations across diverse graph reasoning and real-world tasks of node, link, and graph-levels highlight the superiority of the Hierarchical Language Model for Graph (HLM-G), which employs a two-block architecture to capture node-centric local information and interaction-centric global structure.

Abstract

Large language models (LLMs) are being increasingly explored for graph tasks. Despite their remarkable success in text-based tasks, LLMs' capabilities in understanding explicit graph structures remain limited, particularly with large graphs. In this work, we introduce Hierarchical Language Model for Graph (HLM-G), which employs a two-block architecture to capture node-centric local information and interaction-centric global structure, effectively enhancing graph structure understanding abilities. The proposed scheme allows LLMs to address various graph queries with high efficacy, efficiency, and robustness, while reducing computational costs on large-scale graph tasks. Furthermore, we demonstrate the interpretability of our model using intrinsic attention weights and established explainers. Comprehensive evaluations across diverse graph reasoning and real-world tasks of node, link, and graph-levels highlight the superiority of our method, marking a significant advancement in the application of LLMs to graph understanding.

Figures (6)

• Figure 1: Hierarchical Model Design: Local Block employs intra-node attention to learn local node and structural features. Pooling layer combines these features and Global Block utilizes inter-node attention to capture higher-level interactions, enabling comprehensive graph understanding. The Hierarchical model design results in a model which is highly scalable and delivers robust performance across both structure reasoning tasks and real world graph prediction tasks. The model also supports dual interpretability: node-level interpretability through the Global Block and fine-grained token-level interpretability via the Local Block, making it not only powerful but also transparent in its predictions.
• Figure 2: Explainer Based Interpretation Comparisons. This figure illustrates the interpretability performance of BERT, GIN, and our method on 4 graph reasoning datasets with reasoning ground truths. $k$ indicates the $k$ most important nodes that interpreted by the model are selected.
• Figure 3: Layer-by-Layer Attention Interpretation. This figure compares the mean attention scores for relevant nodes with irrelevant nodes across each layer of the model in 4 graph reasoning tasks. The increased scores in higher layers emphasizes the model's capability to learn larger scale structure information and identify relevant graph nodes effectively.
• Figure 4: Fidelity results. This figure measures the faithfulness of 4 explainers to 3 models using Fidelity scores across different Sparsities. Results should be compared across different explainers within the same dataset and method.
• Figure 5: Interpretation visualization of HLM-G (ours) on the node degree dataset.