Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Aggregators in Neural Algorithmic Reasoning

Kaijia Xu, Petar Veličković

TL;DR

This work investigates neural algorithmic reasoning beyond permutation-invariant graph neural networks by introducing RNAR, a recurrent aggregator based on LSTMs that processes messages in a fixed node order. By leveraging natural ordering in sequential algorithms, RNAR achieves strong results, including a state-of-the-art mean micro-F1 on Quickselect (87.08%) and improvements across multiple CLRS-30 tasks. The study demonstrates that non-commutative, order-aware aggregators can substantially boost performance, while also outlining trade-offs in memory usage and computation. The findings suggest a broader potential for non-invariant aggregators in neural algorithmic reasoning and invite further exploration of ordering, memory-efficient recurrent designs, and alignment with automata-like reasoning.

Abstract

Neural algorithmic reasoning (NAR) is an emerging field that seeks to design neural networks that mimic classical algorithmic computations. Today, graph neural networks (GNNs) are widely used in neural algorithmic reasoners due to their message passing framework and permutation equivariance. In this extended abstract, we challenge this design choice, and replace the equivariant aggregation function with a recurrent neural network. While seemingly counter-intuitive, this approach has appropriate grounding when nodes have a natural ordering -- and this is the case frequently in established reasoning benchmarks like CLRS-30. Indeed, our recurrent NAR (RNAR) model performs very strongly on such tasks, while handling many others gracefully. A notable achievement of RNAR is its decisive state-of-the-art result on the Heapsort and Quickselect tasks, both deemed as a significant challenge for contemporary neural algorithmic reasoners -- especially the latter, where RNAR achieves a mean micro-F1 score of 87%.

Recurrent Aggregators in Neural Algorithmic Reasoning

TL;DR

This work investigates neural algorithmic reasoning beyond permutation-invariant graph neural networks by introducing RNAR, a recurrent aggregator based on LSTMs that processes messages in a fixed node order. By leveraging natural ordering in sequential algorithms, RNAR achieves strong results, including a state-of-the-art mean micro-F1 on Quickselect (87.08%) and improvements across multiple CLRS-30 tasks. The study demonstrates that non-commutative, order-aware aggregators can substantially boost performance, while also outlining trade-offs in memory usage and computation. The findings suggest a broader potential for non-invariant aggregators in neural algorithmic reasoning and invite further exploration of ordering, memory-efficient recurrent designs, and alignment with automata-like reasoning.

Abstract

Neural algorithmic reasoning (NAR) is an emerging field that seeks to design neural networks that mimic classical algorithmic computations. Today, graph neural networks (GNNs) are widely used in neural algorithmic reasoners due to their message passing framework and permutation equivariance. In this extended abstract, we challenge this design choice, and replace the equivariant aggregation function with a recurrent neural network. While seemingly counter-intuitive, this approach has appropriate grounding when nodes have a natural ordering -- and this is the case frequently in established reasoning benchmarks like CLRS-30. Indeed, our recurrent NAR (RNAR) model performs very strongly on such tasks, while handling many others gracefully. A notable achievement of RNAR is its decisive state-of-the-art result on the Heapsort and Quickselect tasks, both deemed as a significant challenge for contemporary neural algorithmic reasoners -- especially the latter, where RNAR achieves a mean micro-F1 score of 87%.
Paper Structure (11 sections, 3 equations, 1 figure, 4 tables)

This paper contains 11 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Towards RNAR
  3. Aggregation functions
  4. Why would we ever drop permutation invariance in NARs?
  5. The RNAR architecture
  6. Evaluating RNAR
  7. RNAR ablations on base model and positional information
  8. RNAR gains additional power through architectural choices
  9. RNAR can meaningfully exploit positional features
  10. RNAR's robustness against choice of node permutation
  11. Timing performance of RNAR

Figures (1)

  • Figure 1: A comparison of the wall-clock times ($y$-axis, in seconds) required for completing a certain number of training steps ($x$-axis), for the baseline Triplet-GMPNN (in blue) against RNAR (in orange), across eight representative algorithms in CLRS-30.