Topological Neural Networks: Mitigating the Bottlenecks of Graph Neural Networks via Higher-Order Interactions
Lorenzo Giusti
TL;DR
The paper identifies over-squashing as a core bottleneck in message-passing GNNs when modeling long-range dependencies, and develops a unified theoretical framework linking width, depth, and graph topology to this phenomenon. It then introduces Topological Neural Networks that propagate messages through higher-order structures (simplicial and cell complexes) to decouple computation from the input graph and to capture higher-order interactions. Two attention-based architectures, Simplicial Attention Networks (SAN) and Cell Attention Networks (CAN), along with Enhanced Cellular Isomorphism Networks (CIN++), are proposed to leverage upper/lower adjacencies and ring-like structures for anisotropic, multi-scale information flow. Empirical validation across trajectory data, supramolecular chemistry benchmarks, and large-scale molecular datasets (e.g., ZINC, MOLHIV, Peptides, and TUDataset) demonstrates strong gains over traditional GNNs and state-of-the-art topological methods, illustrating improved modeling of long-range and group interactions with manageable computational overhead. Overall, the work provides a principled, topology-aware framework for scalable, higher-order representation learning with significant implications for chemistry, neuroscience, and physics.
Abstract
The irreducible complexity of natural phenomena has led Graph Neural Networks to be employed as a standard model to perform representation learning tasks on graph-structured data. While their capacity to capture local and global patterns is remarkable, the implications associated with long-range and higher-order dependencies pose considerable challenges to such models. This work starts with a theoretical framework to reveal the impact of network's width, depth, and graph topology on the over-squashing phenomena in message-passing neural networks. Then, the work drifts towards, higher-order interactions and multi-relational inductive biases via Topological Neural Networks. Such models propagate messages through higher-dimensional structures, providing shortcuts or additional routes for information flow. With this construction, the underlying computational graph is no longer coupled with the input graph structure, thus mitigating the aforementioned bottlenecks while accounting also for higher-order interactions. Inspired by Graph Attention Networks, two topological attention networks are proposed: Simplicial and Cell Attention Networks. The rationale behind these architecture is to leverage the extended notion of neighbourhoods provided by the arrangement of groups of nodes within a simplicial or cell complex to design anisotropic aggregations able to measure the importance of the information coming from different regions of the domain. By doing so, they capture dependencies that conventional Graph Neural Networks might miss. Finally, a multi-way communication scheme is introduced with Enhanced Cellular Isomorphism Networks, which augment topological message passing schemes to enable a direct interactions among groups of nodes arranged in ring-like structures.