Mind The Gap: Quantifying Mechanistic Gaps in Algorithmic Reasoning via Neural Compilation
Lucas Saldyt, Subbarao Kambhampati
TL;DR
The paper tackles the challenge of mechanistic faithfulness in neural algorithmic reasoning by proposing a neural-compilation framework that encodes algorithms into Graph Attention Network parameters. It uses BFS, DFS, and Bellman-Ford to compare learned behavior with neurally compiled ground-truth, focusing on internal attention patterns and external trace predictions. Key findings show high predictive accuracy can accompany substantial mechanistic gaps, with no strong correlation between accuracy and faithfulness, even for algorithmically aligned tasks. The work highlights the scalar bottleneck and training dynamics as sources of underconvergence and suggests curriculum and architectural adjustments to foster more faithful algorithmic reasoning in neural models.
Abstract
This paper aims to understand how neural networks learn algorithmic reasoning by addressing two questions: How faithful are learned algorithms when they are effective, and why do neural networks fail to learn effective algorithms otherwise? To answer these questions, we use neural compilation, a technique that directly encodes a source algorithm into neural network parameters, enabling the network to compute the algorithm exactly. This enables comparison between compiled and conventionally learned parameters, intermediate vectors, and behaviors. This investigation is crucial for developing neural networks that robustly learn complexalgorithms from data. Our analysis focuses on graph neural networks (GNNs), which are naturally aligned with algorithmic reasoning tasks, specifically our choices of BFS, DFS, and Bellman-Ford, which cover the spectrum of effective, faithful, and ineffective learned algorithms. Commonly, learning algorithmic reasoning is framed as induction over synthetic data, where a parameterized model is trained on inputs, traces, and outputs produced by an underlying ground truth algorithm. In contrast, we introduce a neural compilation method for GNNs, which sets network parameters analytically, bypassing training. Focusing on GNNs leverages their alignment with algorithmic reasoning, extensive algorithmic induction literature, and the novel application of neural compilation to GNNs. Overall, this paper aims to characterize expressability-trainability gaps - a fundamental shortcoming in learning algorithmic reasoning. We hypothesize that inductive learning is most effective for parallel algorithms contained within the computational class \texttt{NC}.