Synthesizer Based Efficient Self-Attention for Vision Tasks

TL;DR

This work addresses the heavy computational cost and structural disruption of dot-product self-attention in vision tasks by introducing Synthesizing Tensor Transformations (STT), a family of tensor-based plug-ins that learn synthetic attention weights directly from image tensors. The authors develop several STT variants, including Tensor Dense, Tensor Random, and their Kronecker-factorized forms, plus a Mixture Tensor Synthesizer, to reduce parameter counts and improve robustness. Empirical results on CIFAR and COCO demonstrate competitive accuracy and enhanced robustness to Gaussian noise, flips, and rotations, along with improved efficiency and scalability. Overall, STT offers a principled, extensible approach to vision self-attention that preserves tensor structure and reduces computation, with strong potential for broader adoption in vision transformers and related architectures.

Abstract

Self-attention module shows outstanding competence in capturing long-range relationships while enhancing performance on vision tasks, such as image classification and image captioning. However, the self-attention module highly relies on the dot product multiplication and dimension alignment among query-key-value features, which cause two problems: (1) The dot product multiplication results in exhaustive and redundant computation. (2) Due to the visual feature map often appearing as a multi-dimensional tensor, reshaping the scale of the tensor feature to adapt to the dimension alignment might destroy the internal structure of the tensor feature map. To address these problems, this paper proposes a self-attention plug-in module with its variants, namely, Synthesizing Tensor Transformations (STT), for directly processing image tensor features. Without computing the dot-product multiplication among query-key-value, the basic STT is composed of the tensor transformation to learn the synthetic attention weight from visual information. The effectiveness of STT series is validated on the image classification and image caption. Experiments show that the proposed STT achieves competitive performance while keeping robustness compared to self-attention in the aforementioned vision tasks.

Figures (7)

• Figure S1: Dot product attention VS. synthesizers for NLPsynthesizer. (a) Dot product based self-attention for NLP. (b) Dense Synthesizer for NLP. (c) Random Synthesizer for NLP.
• Figure S2: Dot-product visual self-attention.
• Figure S3: This figure visualizes an example of the $n-$mode product. The third-order tensor $\mathcal{X} \in \mathbb{R}^{8\times7\times9}$ is mapped into $\mathcal{Z} \in \mathbb{R}^{4\times5\times6}$ with a 3-mode product using matrices $\mathbf{A}_{(H)} \in \mathbb{R}^{4\times8}$, $\mathbf{A}_{(W)} \in \mathbb{R}^{5\times7}$, $\mathbf{A}_{(C)} \in \mathbb{R}^{6\times9}$.
• Figure S4: Illustration for tensor multiplication with Kronecker procduct operator. In this case, a three-dimensional tensor is first expanded along its vertical dimension. Then, the horizontal dimension is expanded. After that, the third dimension is reduced to get a two-dimensional matrix.
• Figure S5: Dense synthesizers and random synthesizer for vision tasks. (a) Tensor Dense Synthesizer for vision tasks. (b) Tensor Random Synthesizer for vision tasks.

Theorems & Definitions (2)

• Definition 1: $n-$mode product
• Definition 2: Tensor transformation