DARTS-GT: Differentiable Architecture Search for Graph Transformers with Quantifiable Instance-Specific Interpretability Analysis
Shruti Sarika Chakraborty, Peter Minary
TL;DR
DARTS-GT introduces a differentiable NAS framework to enable depth-wise heterogeneity in Graph Transformer attention by making queries depend on node features while keys/values come from a layer-specific GNN, optimized with DARTS. It couples this architectural design with a novel, quantitative interpretability framework based on causal ablation (Head Deviation, Specialization, Focus) to assess which heads and nodes drive predictions, at both instance and dataset levels. Empirically, DARTS-GT achieves state-of-the-art results on several benchmarks and reveals dataset-specific architecture patterns, while also showing that attention salience often does not align with causal importance, underlining the value of causal interpretability. Overall, the work demonstrates that Graph Transformers can achieve enhanced performance without sacrificing interpretability by rethinking the attention mechanism and employing principled NAS.
Abstract
Graph Transformers (GTs) have emerged as powerful architectures for graph-structured data, yet remain constrained by rigid designs and lack quantifiable interpretability. Current state-of-the-art GTs commit to fixed GNN types across all layers, missing potential benefits of depth-specific component selection, while their complex architectures become opaque where performance gains cannot be distinguished between meaningful patterns and spurious correlations. We redesign GT attention through asymmetry, decoupling structural encoding from feature representation: queries derive from node features while keys and values come from GNN transformations. Within this framework, we use Differentiable ARchiTecture Search (DARTS) to select optimal GNN operators at each layer, enabling depth-wise heterogeneity inside transformer attention itself (DARTS-GT). To understand discovered architectures, we develop the first quantitative interpretability framework for GTs through causal ablation. Our metrics (Head-deviation, Specialization, and Focus), identify which heads and nodes drive predictions while enabling model comparison. Experiments across eight benchmarks show DARTS-GT achieves state-of-the-art on four datasets while remaining competitive on others, with discovered architectures revealing dataset-specific patterns. Our interpretability analysis reveals that visual attention salience and causal importance do not always correlate, indicating widely used visualization approaches may miss components that actually matter. Crucially, heterogeneous architectures found by DARTS-GT consistently produced more interpretable models than baselines, establishing that Graph Transformers need not choose between performance and interpretability.