Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

TL;DR

The paper addresses the challenge of transforming Transformers to handle inputs longer than their training context. It foregrounds positional encoding as the key factor for length extrapolation and organizes techniques into interpolation and extrapolation, further split into zero-shot and fine-tuned families. Key contributions include a comprehensive taxonomy, detailed surveys of ALiBi, RoPE, XPOS, LongNet, Landmark Attention, and memory-augmented methods, plus critical analysis of zero-shot and fine-tuned extrapolation strategies and their empirical performance. The work highlights practical implications for long-document understanding and retrieval-augmented generation, identifying promising directions for scalable, memory-efficient long-context modeling in real-world applications.

Abstract

Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they cannot perform length extrapolation to handle long sequences, which severely hinders their application in scenarios demanding long input sequences such as legal or scientific documents. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, we aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research.

Figures (17)

• Figure 1: Taxonomy for context length extension techniques in LLMs. The figure distinguishes the techniques into interpolation and extrapolation, where they are further classified into zero-shot and fine-tuned branches. Positional encoding, Retrieval, Attention and RoPE based techniques are explored the most in this domain of context length extension.
• Figure 2: Implementation of ALiBi press2021train. When calculating attention in a neural network, the figure's method involves adding a fixed bias to each attention score before applying the softmax function. This bias is the same for all attention scores in a specific head. The rest of the computation remains unchanged. The variable 'm' is a constant specific to each attention head and is set without being adjusted during training. This approach works well across different types of text, various models, and different computational resources.
• Figure 3: Visualization of RoPE SU2024127063, which employs rotational matrices to capture precise absolute positional information in token sequences. By rotating segments of query and key projection matrices at different speeds, RoPE ensures unique rotations, influencing attention scores. The figure visually explains this innovative approach, emphasizing RoPE's reliance on relative distances for improved token relationship comprehension in self-attention models.
• Figure 4: Implementation of Randomized Positional Encodings ruoss2023randomized. When testing a model with longer input sequences, the typical way of adding position information can lead to values that were not seen during training. The figure's solution is to address this issue by assigning a random (or ordered) positional encoding vector that covers the entire range of possible positions during testing to each training example.
• Figure 5: Implementation of block-wise Causal Attention, which is trained on short texts similar to regular Transformers, using causal masking. For longer sequences during testing, blockwise causal attention is employed, which efficiently reuses overlapping parts like key and value vectors. sun2022length