Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Algorithms Align with Neural Execution

Valerie Engelmayer, Dobrik Georgiev, Petar Veličković

TL;DR

This work argues that neural algorithmic reasoning should leverage parallel computation, as parallel algorithms align with the intrinsic parallelism of neural architectures and can reduce training time and improve predictions. Using the CLRS neural framework, the paper analyzes parallel versus sequential implementations of searching, sorting, and strongly connected components, showing that parallel approaches yield shorter, less redundant computation trajectories and higher neural efficiency. The results demonstrate that parallel algorithms are easier to learn and execute in neural models, with significant training speedups and improved generalization, particularly for out-of-distribution data. The study connects neural computation with classical parallel models (Processor Arrays, PRAM) and highlights practical implications for designing neural learners that exploit parallelism in graph-based processing. Overall, adopting parallel algorithms enhances training efficiency and predictive performance, guiding future integration of parallel algorithmic reasoning in neural architectures.

Abstract

Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve (often strongly) superior predictive performance.

Parallel Algorithms Align with Neural Execution

TL;DR

This work argues that neural algorithmic reasoning should leverage parallel computation, as parallel algorithms align with the intrinsic parallelism of neural architectures and can reduce training time and improve predictions. Using the CLRS neural framework, the paper analyzes parallel versus sequential implementations of searching, sorting, and strongly connected components, showing that parallel approaches yield shorter, less redundant computation trajectories and higher neural efficiency. The results demonstrate that parallel algorithms are easier to learn and execute in neural models, with significant training speedups and improved generalization, particularly for out-of-distribution data. The study connects neural computation with classical parallel models (Processor Arrays, PRAM) and highlights practical implications for designing neural learners that exploit parallelism in graph-based processing. Overall, adopting parallel algorithms enhances training efficiency and predictive performance, guiding future integration of parallel algorithmic reasoning in neural architectures.

Abstract

Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve (often strongly) superior predictive performance.
Paper Structure (32 sections, 1 theorem, 9 equations, 7 figures, 4 tables)

This paper contains 32 sections, 1 theorem, 9 equations, 7 figures, 4 tables.

Table of Contents

  1. Motivation
  2. Parallel Computing
  3. Parallel Computational Models
  4. Processor Arrays.
  5. PRAM Models.
  6. Efficiency
  7. Examples of Parallel Algorithms
  8. Searching.
  9. Sorting.
  10. Strongly Connected Components.
  11. Analogy to Neural Networks
  12. Efficiency of Executing Algorithms Neurally
  13. Parallel Algorithms Entail Higher Efficiency
  14. Shorter Trajectories.
  15. Less Redundancy.
  16. ...and 17 more sections

Key Result

Corollary 1

Let $\mathcal{G}$ be a scalable GNN operating over a graph with $n$ nodes and $m$ edges. Further let $\mathcal{S}$ be a sequential, and $\mathcal{P}$ an efficient parallel algorithm on $n$ processors, both of complexity $C$. Then executing $\mathcal{S}$ and $\mathcal{P}$ on $\mathcal{G}$, respective

Figures (7)

  • Figure 1: Trajectories of sequential and parallel algorithms, as well as neural processing.
  • Figure 2: Local view on information flow in different computational models at two different time steps $t$ and $\Tilde{t}$.
  • Figure 3: Necessary information flow when searching $x$ in $A = [A_0, \dots, A_4]$ using different algorithms. Active nodes and edges in color.
  • Figure 4: Consecutive steps of passing the source node index $s$ during a BFS of DCSC. Note how repeating the computation would not change the state of the rightmost node, so redundant computations do not require to be parameterised differently.
  • Figure 5: Training times of sequential algorithms with samples of input size $n=16$, relative to their respective parallel counterparts.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Definition 1
  • Definition 2
  • Definition 3
  • Corollary 1
  • proof