SHAPE: Shifted Absolute Position Embedding for Transformers

TL;DR

SHAPE targets shift invariance for position embeddings in Transformers by randomly shifting absolute position indices during training. It achieves this with no architectural changes and minimal overhead compared to APE, while remaining competitive with or surpassing RPE in several settings. The experiments on WMT English-German demonstrate SHAPE's strong extrapolation to longer sequences and improved interpolation for rare tokens. Overall, SHAPE provides a simple, effective drop-in replacement for APE that enhances generalization and efficiency.

Abstract

Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.

Figures (9)

• Figure 1: Overview of position representations. (a) APE and (c) SHAPE consider absolute positions in the input layer, whereas (b) RPE considers the relative position of a given token pair in the self-attention mechanism.
• Figure 2: Cosine similarities of the encoder hidden states with different offsets $k \in \{0, 100, 250, 500\}$. Only the representation of SHAPE is invariant with $k$.
• Figure 3: BLEU score improvement from APE on validation and test sets with respect to the source sequence length. The gray color means no training data.
• Figure 4: Tokenwise analysis on gold references: the value in each cell represents the ratio that SHAPE assigns a higher score to a gold token than APE.
• Figure 5: Distribution of source sequence length of each dataset.