End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking

TL;DR

The paper addresses the challenge of algorithmic extrapolation by identifying overthinking as a key barrier in recurrent networks and proposing two innovations: DT-Recall architectures that preserve the problem input within the recurrent loop, and a Progressive Incremental Training routine that maintains iteration-agnostic improvements. Together, these enable extreme extrapolation across prefix sums, maze solving, and chess puzzles, achieving accurate solutions after thousands of iterations and converging to fixed points. The work systematically analyzes robustness to perturbations and input changes, showing that recall prevents forgetting and overthinking, while the progressive loss encourages steady progress. This approach advances scalable reasoning in neural networks with practical implications for solving large, complex problems without retraining. Key contributions include memory-preserving recall connections, a randomized, truncated backpropagation-style training regime with progressive loss, and comprehensive demonstrations of extrapolation capabilities beyond training regimes, plus analyses of convergence behavior and robustness.

Abstract

Machine learning systems perform well on pattern matching tasks, but their ability to perform algorithmic or logical reasoning is not well understood. One important reasoning capability is algorithmic extrapolation, in which models trained only on small/simple reasoning problems can synthesize complex strategies for large/complex problems at test time. Algorithmic extrapolation can be achieved through recurrent systems, which can be iterated many times to solve difficult reasoning problems. We observe that this approach fails to scale to highly complex problems because behavior degenerates when many iterations are applied -- an issue we refer to as "overthinking." We propose a recall architecture that keeps an explicit copy of the problem instance in memory so that it cannot be forgotten. We also employ a progressive training routine that prevents the model from learning behaviors that are specific to iteration number and instead pushes it to learn behaviors that can be repeated indefinitely. These innovations prevent the overthinking problem, and enable recurrent systems to solve extremely hard extrapolation tasks.

Figures (20)

• Figure 1: A 'thinking' network trained on $9\times 9$ mazes and their solutions (left) autonomously synthesizes a scalable algorithm. By running this algorithm for longer, it reliably solves problems of size $59 \times 59$ (middle), $201 \times 201$ (right), and much larger (appendix) without retraining. Standard architectures, and even existing primitive thinking models, fail to tolerate this domain shift.
• Figure 2: Architecture schematics. Left to right: A feed-forward network, a network containing three recurrent blocks (in green) that share weights, and a recurrent network with recall.
• Figure 3: Prefix sum models trained on 32-bit inputs extrapolate to 512-bit data. The value of our recall and progressive loss is clear by how quickly and accurately our models solve this very large problem.
• Figure 4: Maze solving models trained on $\mathbf{9\times9}$ inputs extrapolate to $\mathbf{59\times59}$ problems. Mazes this large cannot be solved without recall, and furthermore progressive loss leads to more accurate models.
• Figure 5: Chess models trained on the first 600K easiest puzzles extrapolate to 600K-700K. Recall and progressive loss are required to retain accuracy with many iterations.