Extensions and Limitations of the Neural GPU

TL;DR

The paper investigates how the Neural GPU generalizes to inputs longer than training data, focusing on model size, curriculum design, and input representations. It shows that larger models and well-structured curricula expand the range of learnable arithmetic tasks (including decimal arithmetic) but reveals persistent failures on highly structured inputs, suggesting incomplete learning of the underlying algorithms. The study emphasizes memory challenges and explores global-data-movement ideas, finding limited universal benefits. Overall, it highlights the need for additional information sources or architectural innovations to achieve robust, true algorithm induction with neural methods.

Abstract

The Neural GPU is a recent model that can learn algorithms such as multi-digit binary addition and binary multiplication in a way that generalizes to inputs of arbitrary length. We show that there are two simple ways of improving the performance of the Neural GPU: by carefully designing a curriculum, and by increasing model size. The latter requires a memory efficient implementation, as a naive implementation of the Neural GPU is memory intensive. We find that these techniques increase the set of algorithmic problems that can be solved by the Neural GPU: we have been able to learn to perform all the arithmetic operations (and generalize to arbitrarily long numbers) when the arguments are given in the decimal representation (which, surprisingly, has not been possible before). We have also been able to train the Neural GPU to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of the operands, although these have succeeded only in their binary representation, and not with perfect accuracy. In addition, we gain insight into the Neural GPU by investigating its failure modes. We find that Neural GPUs that correctly generalize to arbitrarily long numbers still fail to compute the correct answer on highly-symmetric, atypical inputs: for example, a Neural GPU that achieves near-perfect generalization on decimal multiplication of up to 100-digit long numbers can fail on $000000\dots002 \times 000000\dots002$ while succeeding at $2 \times 2$. These failure modes are reminiscent of adversarial examples.

Figures (7)

• Figure 1: The Neural GPU.
• Figure 2: Generalization of training error of Neural GPU is highly dependent on initialization seed. Table measures fraction seeds achieved error $<1\%$. Each entry of this matrix involves running experiments with $15$ different seeds on $13$ different tasks.
• Figure 3: Training (top) and test error (bottom) on the decimal multiplication task. (First) models of different sizes trained without curriculum. Only very large models can successfully optimize the training error, and none are able to generalize. (Second) variable-sized models trained with curriculum over different bases. For instance, we find that a medium-sized model generalizes to much longer test cases on decimal multiplication if it is first trained on binary multiplication and then on quaternary multiplication, before being trained on decimal multiplication. (Third) the best results are achieved when we apply curriculum on models with large numbers of filters. We find that training a model first on binary numbers, then 4-ary, then decimal works significantly better than training it on binary numbers, then 5-ary, then decimal. The difference is comparable to the difference between 128 and 256 filters.
• Figure 4: Influence of curriculum on 3-numbers multiplication task.
• Figure 5: Task of learning binary arithmetic on multiple numbers simultaneously using the operators $+,-,\times,\div$.