Positional Description Matters for Transformers Arithmetic

TL;DR

The paper tackles why Transformers falter on arithmetic tasks due to overreliance on absolute positional information and proposes two broad remedies: modify positional encoding through randomized embedding and reframe data formats to reduce positional bias. It demonstrates strong results across three avenues—large-number multiplication with padding and reversed outputs (achieving up to $15\times 15$ accuracy), length extrapolation via data-format tweaks and alternative embeddings, and integration of arithmetic into natural language contexts. Key findings include that padding and reversed product dramatically improve multiplication, random spaces and increased per-digit information enhance extrapolation, and random embeddings help fuse arithmetic with language prompts. These insights show that even relatively small models can acquire robust arithmetic capabilities when training data and positional cues are structured to emphasize actual arithmetic rules rather than memorized position patterns.

Abstract

Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarkable coding abilities. We observe that a crucial challenge is their naive reliance on positional information to solve arithmetic problems with a small number of digits, leading to poor performance on larger numbers. Herein, we delve deeper into the role of positional encoding, and propose several ways to fix the issue, either by modifying the positional encoding directly, or by modifying the representation of the arithmetic task to leverage standard positional encoding differently. We investigate the value of these modifications for three tasks: (i) classical multiplication, (ii) length extrapolation in addition, and (iii) addition in natural language context. For (i) we train a small model on a small dataset (100M parameters and 300k samples) with remarkable aptitude in (direct, no scratchpad) 15 digits multiplication and essentially perfect up to 12 digits, while usual training in this context would give a model failing at 4 digits multiplication. In the experiments on addition, we use a mere 120k samples to demonstrate: for (ii) extrapolation from 10 digits to testing on 12 digits numbers while usual training would have no extrapolation, and for (iii) almost perfect accuracy up to 5 digits while usual training would be correct only up to 3 digits (which is essentially memorization with a training set of 120k samples).

Figures (2)

• Figure 1: Comparison of pretrained model and trained from the scratch model with and without absolute positional embedding on 100 regular testing samples and repetitive samples. We use pretrained and random initialized GPT2-small with/without the positional embedding and fine-tune/train for 10 epochs with a learning rate 2e-5.
• Figure 2: Comparison of trained from scratch model with and without hash embedding on 100 regular testing samples and repetitive samples. We use random initialized GPT2-small (124M) without the positional embedding and train for 25 epochs with a learning rate 1e-5.