Arbitrary-Length Generalization for Addition in a Tiny Transformer

TL;DR

The paper tackles the challenge of enabling transformers to generalize addition to numbers of unseen lengths. It introduces a two-type training-instance framework combined with autoregressive, right-to-left token generation that mimics manual carry propagation, without using scratchpads. The approach defines first-type (two single-digit addends, no scratchpad) and second-type (previous outputs with carry information) training instances, enabling robust generalization up to $n$ digits, with reported accuracy up to $n=1000$ digits and reproducible results via an R implementation. This method offers a scalable route to learn long-sequence arithmetic and could be extended to more complex operations or repeated procedural tasks.

Abstract

This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/.

Figures (1)

• Figure 1: Masking attention for the concatenated input/output. The gray area represents the unmask region.