Vision Transformers are Circulant Attention Learners

TL;DR

This work tackles the quadratic complexity of self-attention in vision Transformers by uncovering a near-BCCB structure in attention maps and enforcing a Block Circulant with Circulant Blocks (BCCB) form. It proposes Circulant Attention, which replaces dense attention with a BCCB-based mechanism computable in $O(N\log N)$ via 2D FFT, while preserving much of the expressive power of vanilla attention. A token reweighting module enhances capacity, and the approach serves as a plug-in for DeiT, PVT, and Swin, achieving strong results on ImageNet, COCO, and ADE20K with substantial efficiency gains. The method offers a practical alternative for scalable, high-resolution vision Transformers and is supported by code for replication and extension.

Abstract

The self-attention mechanism has been a key factor in the advancement of vision Transformers. However, its quadratic complexity imposes a heavy computational burden in high-resolution scenarios, restricting the practical application. Previous methods attempt to mitigate this issue by introducing handcrafted patterns such as locality or sparsity, which inevitably compromise model capacity. In this paper, we present a novel attention paradigm termed \textbf{Circulant Attention} by exploiting the inherent efficient pattern of self-attention. Specifically, we first identify that the self-attention matrix in vision Transformers often approximates the Block Circulant matrix with Circulant Blocks (BCCB), a kind of structured matrix whose multiplication with other matrices can be performed in $\mathcal{O}(N\log N)$ time. Leveraging this interesting pattern, we explicitly model the attention map as its nearest BCCB matrix and propose an efficient computation algorithm for fast calculation. The resulting approach closely mirrors vanilla self-attention, differing only in its use of BCCB matrices. Since our design is inspired by the inherent efficient paradigm, it not only delivers $\mathcal{O}(N\log N)$ computation complexity, but also largely maintains the capacity of standard self-attention. Extensive experiments on diverse visual tasks demonstrate the effectiveness of our approach, establishing circulant attention as a promising alternative to self-attention for vision Transformer architectures. Code is available at https://github.com/LeapLabTHU/Circulant-Attention.

Figures (5)

• Figure 1: An illustration of vanilla self-attention and the proposed circulant attention. The $\sigma$ represents Softmax function, $\odot$ is the Hadamard product, and $\mathcal{F}_{\rm 2D}, \mathcal{F}_{\rm 2D}^{-1}$ denote the 2D discrete Fourier transform (DFT) and its inversion, respectively. Our circulant attention largely inherits the paradigm of self-attention, except for employing BCCB attention matrices. This simple modification enables our design to be efficiently calculated through DFT-based multiplication, thereby achieving $\mathcal{O}(N\log N)$ complexity. The scaling factor and head-wise summation are omitted for simplicity. Please refer to \ref{['sec:method']} for details.
• Figure 2: $N\!\times\!N$ attention maps from DeiT deit, where $N=H\!\times\!W, H=W=14$. Without handcrafted constraints, the self-attention module in vision Transformer learns near BCCB patterns. For better observation, we divide the matrix into $H\!\times\!H$ blocks of shape $W\!\times\!W$ using gray dashed lines. Zoom in for best view.
• Figure 3: The $N\!\times\!N$ attention matrices and $H\!\times\!W$ attention distributions of consecutive queries, where $N=H\!\times\!W, H=W=14$. The red points correspond to the query tokens. When exhibiting BCCB pattern, the attention distributions show convolution-like translation invariance.
• Figure 4: Comparisons between self-attention and the proposed circulant attention in (a) FLOPs, (b) inference FPS, and (c, d) throughput-accuracy trade-off. Throughput is measured on a RTX3090 GPU.
• Figure 5: An illustration of the equivalent global convolution kernels from CA-DeiT model.