GraphGhost: Tracing Structures Behind Large Language Models

TL;DR

GraphGhost introduces a graph-based framework that represents neuron activations and their signal propagation as attribution graphs, aggregated into a unified GraphGhost to study reasoning in decoder-only LLMs. By applying graph algorithms such as PageRank and conducting targeted structural interventions (e.g., muting nodes or editing edges), the approach reveals both shared and model-specific reasoning patterns across diverse datasets and models, and demonstrates causal influence of key neurons on semantic and logical flow. The authors provide case studies and quantitative analyses showing how perturbations to tokens or graph structure can alter reasoning trajectories, including potential prompting and domain generalization implications. Across GSM8K, MAWPS, ProntoQA, BoolQA, ARC-Easy, and QASC, GraphGhost yields insights into semantic merging, logic shifts, and the distribution of influential tokens, establishing a principled framework for analyzing, debugging, and potentially guiding LLM reasoning. Overall, GraphGhost offers a rigorous, graph-centric tool for understanding the structural foundations of reasoning in LLMs and for exploring targeted interventions to shape their outputs.

Abstract

Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, yet the structural mechanisms underlying these abilities remain under explored. In this work, we introduce GraphGhost, a unified framework that represents neuron activations and their signal propagation as graphs, explaining how LLMs capture structural semantics from sequential inputs and generate outputs through structurally consistent mechanisms. This graph-based perspective enables us to employ graph algorithms such as PageRank to characterize the properties of LLMs, revealing both shared and model-specific reasoning behaviors across diverse datasets. We further identify the activated neurons within GraphGhost and evaluate them through structural interventions, showing that edits to key neuron nodes can trigger reasoning collapse, altering both logical flow and semantic understanding. Together, these contributions position GraphGhost as a powerful tool for analyzing, intervening in, and ultimately understanding the structural foundations of reasoning in LLMs.

Figures (12)

• Figure 1: An illustration of GraphGhost based on Qwen3-0.6B. a) GraphGhost illustrates a structural mechanism within LLMs that operates consistently across datasets. L indicates the layer number. Sequential inputs activate subsets of neurons, which merge tokens in the intermediate layers to capture semantic meaning. These activated neurons then propagate signals to the logit layers to generate outputs. b) Muting the neuron node. Changing the structure of GraphGhost leads to corresponding changes in both language expression and logical reasoning. "等于" means "equal to" in Chinese.
• Figure 2: A toy example of path reasoning on a graph. (a) The simplified descriptions to predict the path between 4 and 7; (b) The circuit tracing for the interpretation of why token 5 is selected to be predicted.
• Figure 3: Case study of tracing contributing tokens for the answer 14 in Qwen3-0.6B model. We highlight the numerical reasoning flow captured by GraphGhost to visualize how number-related tokens contribute to the final prediction.
• Figure 4: A subgraph of GraphGhost in Qwen3-0.6B model. We include 4 datasets from different domains.
• Figure 5: The top merging nodes across the models and datasets.