Topologic Attention Networks: Attending to Direct and Indirect Neighbors through Gaussian Belief Propagation

TL;DR

Topologic Attention Networks address the limitation of traditional GNNs in modeling long-range dependencies by introducing a probabilistic topologic attention mechanism based on Gaussian Belief Propagation. TANs learn how information should flow through a graph's topology via a learnable precision matrix $J$ and evidence vector $h$, enabling dynamic neighborhoods that integrate both direct and indirect influences. The framework offers three precision designs (Pairwise Normal, Diagonally Dominant, Laplacian), a multi-head GaBP extension, and an implicit-differentiation training scheme to manage memory, delivering state-of-the-art or competitive results on six node-classification benchmarks. The work highlights that partial convergence of GaBP can still yield high accuracy, and points toward scalable inference via hierarchical solvers as a key direction for future development.

Abstract

Graph Neural Networks rely on local message passing, which limits their ability to model long-range dependencies in graphs. Existing approaches extend this range through continuous-time dynamics or dense self-attention, but both suffer from high computational cost and limited scalability. We propose Topologic Attention Networks, a new framework that applies topologic attention, a probabilistic mechanism that learns how information should flow through both direct and indirect connections in a graph. Unlike conventional attention that depends on explicit pairwise interactions, topologic attention emerges from the learned information propagation of the graph, enabling unified reasoning over local and global relationships. This method achieves provides state-of-the-art performance across all measured baseline models. Our implementation is available at https://github.com/Marshall-Rosenhoover/Topologic-Attention-Networks.

Figures (11)

• Figure 1: Illustration of a single head of topologic attention via a learned Gaussian Graphical Model, where node shading encodes confidence and edge color denotes interaction polarity and strength.
• Figure 2: Architecture of the Topologic Attention Layer.
• Figure 3: Depiction of how to construct a learned precision matrix $J$.
• Figure 4: Training Over Time of the Diagonally Dominant Precision Matrix on the Wisconsin Dataset.
• Figure 5: Training Over Time of the Laplacian Precision Matrix on the Wisconsin Dataset.