Efficient Attention Mechanisms for Large Language Models: A Survey

TL;DR

The paper tackles the efficiency bottleneck of self-attention in Transformers by surveying two principal approaches: Linear Attention, which achieves linear or near-linear complexity through kernel approximations, recurrence, and fast-weight dynamics, and Sparse Attention, which reduces computation by constraining attention to selected token subsets. It classifies and analyzes a broad range of methods, including kernelized and forgetting-mechanism variants for linear attention, in-context learning interpretations, fixed-pattern and block/ clustering-based sparsity, and system-level hardware considerations. It also examines how these ideas are integrated into large pretrained LLMs through uniform and hybrid designs, highlighting architectural trade-offs and engineering challenges. The survey emphasizes the need for hardware-algorithm co-design, hybrid architectures to balance global and local context, and mechanisms to achieve long-context capabilities with practical deployment impact. Collectively, the work provides a foundational reference for building scalable, efficient language models capable of handling extended contexts while maintaining performance.

Abstract

Transformer-based architectures have become the prevailing backbone of large language models. However, the quadratic time and memory complexity of self-attention remains a fundamental obstacle to efficient long-context modeling. To address this limitation, recent research has introduced two principal categories of efficient attention mechanisms. Linear attention methods achieve linear complexity through kernel approximations, recurrent formulations, or fastweight dynamics, thereby enabling scalable inference with reduced computational overhead. Sparse attention techniques, in contrast, limit attention computation to selected subsets of tokens based on fixed patterns, block-wise routing, or clustering strategies, enhancing efficiency while preserving contextual coverage. This survey provides a systematic and comprehensive overview of these developments, integrating both algorithmic innovations and hardware-level considerations. In addition, we analyze the incorporation of efficient attention into largescale pre-trained language models, including both architectures built entirely on efficient attention and hybrid designs that combine local and global components. By aligning theoretical foundations with practical deployment strategies, this work aims to serve as a foundational reference for advancing the design of scalable and efficient language models.

Figures (4)

• Figure 1: Taxonomy of Efficient Attention Mechanisms.
• Figure 2: Dual form of Linear Attention.
• Figure 3: Block-sparse attention: the long sequence is divided into several blocks, and each token attends only to its local window and the top-k related blocks.
• Figure 4: Architecture of different stacked models.