Reasoning Algorithmically in Graph Neural Networks

TL;DR

This work investigates Neural Algorithmic Reasoning (NAR) as a principled approach to endow Graph Neural Networks with algorithmic reasoning capabilities.By linking neural models with tropical algebra, duality, and the Encode-Process-Decode framework, the author shows that GNs can approximate min-aggregated dynamic programming and learn to execute classical graph algorithms with arbitrary precision. The dissertation demonstrates practical benefits across planning (learned heuristics for A*), max-flow/min-cut tasks via Dual Algorithmic Reasoning, and combinatorial optimization (TSP/VKC) through transfer of algorithmic priors. Empirical results include planning improvements, edge-classification on Brain Vessel Graph benchmarks, and NP-hard CO problem approximations, highlighting both the potential and limitations of algorithmically informed neural models. Overall, the thesis provides theoretical connections, architectural guidelines, and empirical evidence that algorithmic priors can enhance OOD generalization and enable scalable neural execution of algorithms on large graphs.

Abstract

The development of artificial intelligence systems with advanced reasoning capabilities represents a persistent and long-standing research question. Traditionally, the primary strategy to address this challenge involved the adoption of symbolic approaches, where knowledge was explicitly represented by means of symbols and explicitly programmed rules. However, with the advent of machine learning, there has been a paradigm shift towards systems that can autonomously learn from data, requiring minimal human guidance. In light of this shift, in latest years, there has been increasing interest and efforts at endowing neural networks with the ability to reason, bridging the gap between data-driven learning and logical reasoning. Within this context, Neural Algorithmic Reasoning (NAR) stands out as a promising research field, aiming to integrate the structured and rule-based reasoning of algorithms with the adaptive learning capabilities of neural networks, typically by tasking neural models to mimic classical algorithms. In this dissertation, we provide theoretical and practical contributions to this area of research. We explore the connections between neural networks and tropical algebra, deriving powerful architectures that are aligned with algorithm execution. Furthermore, we discuss and show the ability of such neural reasoners to learn and manipulate complex algorithmic and combinatorial optimization concepts, such as the principle of strong duality. Finally, in our empirical efforts, we validate the real-world utility of NAR networks across different practical scenarios. This includes tasks as diverse as planning problems, large-scale edge classification tasks and the learning of polynomial-time approximate algorithms for NP-hard combinatorial problems. Through this exploration, we aim to showcase the potential integrating algorithmic reasoning in machine learning models.

Figures (18)

• Figure 1: Visual representation of undirected and directed graphs. In (b), node adjacencies $\{\{v_1, v_4\}, \{v_2, v_3\}, \{v_4, v_5\}\}$ are bi-directional connections and may be expressed as undirected edges. Equivalently, all edges in (a) may be replaced by two directed edges.
• Figure 2: Illustration of various graph types. For better clarity, self-loops are not shown. In the grid illustration, $v_1, v_3, v_4$ and $v_6$ possess self-loops, ensuring the graph's 3-regularity.
• Figure 3: This graphic showcases how messages are computed and passed in graph networks. At step $\ell = 0$, we focus on node $v$. To update its representation ${\bm{h}}^{(0)}_v$, we compute messages $m_x = \psi({\bm{h}}^{(0)}_{v},{\bm{h}}^{(0)}_{x}, {\bm{h}}^{(0)}_{xv})$ and $m_z = \psi({\bm{h}}^{(0)}_{v},{\bm{h}}^{(0)}_{z}, {\bm{h}}^{(0)}_{zv})$ from its neighbours $x$ and $z$. ${\bm{h}}^{1}_v$ is then obtained through application of \ref{['eq:gn-mp']}. At step $\ell=1$, $u$ is updated based on its only neighbour $v$, whose has already gathered information from $x$ and $z$ at $\ell=0$. As a result, at a subsequent step $\ell=2$, ${\bm{h}}^{(2)}$ will be conditioned not only on ${\bm{h}}^{(1)}_v$ but also on ${\bm{h}}^{(0)}_x$ and ${\bm{h}}^{(0)}_z$, essentially expanding the range of information $u$ has access to.
• Figure 4: An iterative graph network (above) is compared to a two-layer feedforward graph network (below). Above, node representations are refined via iterative application of the same ${\bm{W}}$, as symbolised by the recurrent connection in the message-passing layer (MP layer). Below, instead, node representations are updated first by application of ${\bm{W}}^{(1)}$ in the first MP block and then updated again by the second MP block symbolised by ${\bm{W}}^{(2)}$. In the picture, $\star$ represents application of \ref{['eq:gn-mp']}.
• Figure 5: Illustration of the algorithmic bottleneck phenomena. Here, we model a shortest path problem in a real-world environment (Natural space) as a graph in the abstract space of algorithms. However, the encoding of multi-dimensional, noisy natural data (i.e., $\tilde{f}$) to a single dimension (i.e., scalars of algorithms) is often performed manually and leads to loss of information harris1955fundamentals. Then, we get a provably correct, but likely suboptimal, solution in the abstract space through applications of a shortest path algorithm (e.g., Dijkstra).

Theorems & Definitions (38)

• Definition 1: Undirected Graph
• Definition 2: Directed Graph
• Definition 3: Outgoing/Ingoing Edges
• Definition 4: Adjacency
• Definition 5: Adjacency Matrix
• Definition 6: Neighbourhood
• Definition 7: Node degree
• Definition 8: Finite Path
• Definition 9: Finite Cycle
• Definition 10: Reachability