Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers
Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, Zhen Zhuang
TL;DR
This work extends neural algorithmic reasoning to hypergraphs using Loop Transformers. It introduces a degradation mechanism that implicitly converts the hypergraph incidence representation into a graph-like form, enabling $O(1)$-layer, $O(1)$-dimensional looped transformers to simulate graph algorithms such as Dijkstra, BFS, and DFS on hypergraphs. It also presents a hyperedge-aware encoding scheme to execute hypergraph-specific procedures like Helly's algorithm, with formal guarantees showing the Transformer can handle high-dimensional, combinatorial data in a structured, iterative manner. The results demonstrate the potential of Loop Transformers as general-purpose algorithmic solvers for hypergraphs, offering theoretical and practical pathways to neural execution of complex combinatorial tasks. The work highlights the transformer’s ability to bridge neural representations with classical algorithms on higher-order structures, paving the way for scalable, neural algorithmic reasoning in domains with rich relational data.
Abstract
Looped Transformers have shown exceptional neural algorithmic reasoning capability in simulating traditional graph algorithms, but their application to more complex structures like hypergraphs remains underexplored. Hypergraphs generalize graphs by modeling higher-order relationships among multiple entities, enabling richer representations but introducing significant computational challenges. In this work, we extend the Loop Transformer architecture's neural algorithmic reasoning capability to simulate hypergraph algorithms, addressing the gap between neural networks and combinatorial optimization over hypergraphs. Specifically, we propose a novel degradation mechanism for reducing hypergraphs to graph representations, enabling the simulation of graph-based algorithms, such as Dijkstra's shortest path. Furthermore, we introduce a hyperedge-aware encoding scheme to simulate hypergraph-specific algorithms, exemplified by Helly's algorithm. We establish theoretical guarantees for these simulations, demonstrating the feasibility of processing high-dimensional and combinatorial data using Loop Transformers. This work highlights the potential of Transformers as general-purpose algorithmic solvers for structured data.