Fastformer: Additive Attention Can Be All You Need

TL;DR

Fastformer introduces a linear-complexity Transformer variant based on additive attention to capture global contexts. By deriving a global query vector and context-aware keys and values through element-wise interactions and additive summarization, it achieves effective long-sequence modeling with reduced computation. Extensive experiments across classification, recommendation, and summarization tasks show competitive accuracy and clear efficiency advantages over both standard Transformers and other linear-complexity variants. The work suggests strong potential for long-document NLP and downstream systems requiring scalable context understanding.

Abstract

Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.

Figures (4)

• Figure 1: The architecture of Fastformer.
• Figure 2: Training and inference speed of different methods. The y-axis (time) is in logarithmic scale.
• Figure 3: Influence of different combination functions.
• Figure 4: Influence of different parameter sharing strategies.