Expressive Power of Graph Transformers via Logic

TL;DR

This work provides formal, logic-based characterizations of graph transformers (GTs) and GPS-networks, analyzing their expressive power under two numerical regimes: real numbers and floating-point numbers. In the real-number setting, GPS-networks are shown to have expressive power equivalent to graded modal logic with a global modality (GML+G) when restricted to FO-definable vertex properties, and GTs align with propositional logic with a global modality (PL+G); in the floating-point setting, GPS-networks match GML with counting global modality (GML+GC) and GTs match PL with counting global modality (PL+GC). The authors introduce global-ratio graded bisimilarity and a van Benthem–Rosen style reduction to prove FO-relative and absolute characterizations, respectively, and extend results to word-shaped graphs and graph classification scenarios. They also analyze the impact of FP arithmetic (underflow, saturation) on expressiveness, show absolute counting becomes possible with FP, and discuss positional encodings and practical implications. Overall, the paper provides foundational, theory-grounded connections between graph neural architectures and logical expressiveness, clarifying what GTs and GPS-networks can and cannot express under realistic numerical regimes. These results offer a principled lens for understanding model capabilities beyond empirical performance, with implications for designing graph models with targeted logical properties.

Abstract

Transformers are the basis of modern large language models, but relatively little is known about their precise expressive power on graphs. We study the expressive power of graph transformers (GTs) by Dwivedi and Bresson (2020) and GPS-networks by Rampásek et al. (2022), both under soft-attention and average hard-attention. Our study covers two scenarios: the theoretical setting with real numbers and the more practical case with floats. With reals, we show that in restriction to vertex properties definable in first-order logic (FO), GPS-networks have the same expressive power as graded modal logic (GML) with the global modality. With floats, GPS-networks turn out to be equally expressive as GML with the counting global modality. The latter result is absolute, not restricting to properties definable in a background logic. We also obtain similar characterizations for GTs in terms of propositional logic with the global modality (for reals) and the counting global modality (for floats).

Theorems & Definitions (108)

• Example 1
• Example 2
• Example 3
• Theorem 4
• Example 5
• Lemma 5
• Example 6
• Lemma 6
• Theorem 7
• Theorem 8