Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures

Alexander M. Fichtl, Jeremias Bohn, Josefin Kelber, Edoardo Mosca, Georg Groh

TL;DR

The paper addresses the computational bottleneck of quadratic attention in transformers and surveys sub-quadratic alternatives, including non-recurrent attention variants, linear RNNs, SSMs, and hybrids. It systematically analyzes time/space complexity, benchmark results, and fundamental limitations, offering a taxonomy, comparative insights, and guidance on where pure attention may still dominate versus where hybrids and memory-centric designs shine. Key contributions include a structured overview of sub-quadratic approaches, a complexity/benchmark synthesis, and an architectural-limitations perspective that contextualizes practical trade-offs for long-context NLP. The findings suggest that while sub-quadratic methods improve efficiency and enable long contexts in edge/mid regimes, they do not yet supersede full attention on frontier-scale tasks; the pragmatic path forward lies in diversified architectures and memory-driven hybrids rather than wholesale transformer replacement.

Abstract

Transformers have dominated sequence processing tasks for the past seven years -- most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. This paper surveys recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze these approaches in terms of compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged.

The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures

TL;DR

The paper addresses the computational bottleneck of quadratic attention in transformers and surveys sub-quadratic alternatives, including non-recurrent attention variants, linear RNNs, SSMs, and hybrids. It systematically analyzes time/space complexity, benchmark results, and fundamental limitations, offering a taxonomy, comparative insights, and guidance on where pure attention may still dominate versus where hybrids and memory-centric designs shine. Key contributions include a structured overview of sub-quadratic approaches, a complexity/benchmark synthesis, and an architectural-limitations perspective that contextualizes practical trade-offs for long-context NLP. The findings suggest that while sub-quadratic methods improve efficiency and enable long contexts in edge/mid regimes, they do not yet supersede full attention on frontier-scale tasks; the pragmatic path forward lies in diversified architectures and memory-driven hybrids rather than wholesale transformer replacement.

Abstract

Transformers have dominated sequence processing tasks for the past seven years -- most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. This paper surveys recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze these approaches in terms of compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged.

Paper Structure

This paper contains 47 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Review Work
  3. Modern $\mathcal{O}(n^2)$ Attention
  4. Reducing KV Cache
  5. Flash Attention
  6. Paged Attention
  7. Sub-Quadratic Architectures
  8. Non-Recurrent $\mathcal{O}(n^{2-\epsilon})$ Attention Variants
  9. Approximate Attention
  10. Sparse Attention
  11. Linear RNN-based Models
  12. Hierarchically Gated Recurrent Neural Network (HGRN)
  13. xLSTM
  14. Lightning Attention
  15. Receptance Weighted Key Value (RWKV)
  16. ...and 32 more sections

Figures (2)

  • Figure 1: The four types of dot-product attention alternatives identified in our survey, including examples for each type. We further distinguish between two major classes for hybrid concepts, namely striped and fusion hybrids, as well as for sub-quadratic attention variants, namely approximate and sparse attention.
  • Figure 2: Different types of hybrids. (a): block types using different primitives are connected in series. (b): block types are connected in parallel.