Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReduceFormer: Attention with Tensor Reduction by Summation

John Yang, Le An, Su Inn Park

TL;DR

This work addresses the bottleneck of self-attention in vision transformers by replacing matrix multiplications and Softmax with simple tensor reductions and element-wise operations. It introduces ReduceFormer, a family of models combining a Multi-Scale Local Context module with a ReduceFormer Attention mechanism that uses global summations to project and relate spatial features, yielding substantially reduced computational cost while preserving accuracy. Empirical results on ImageNet-1K show competitive performance with significant gains in latency and throughput on both edge devices (e.g., Drive Orin) and data-center GPUs (L40), outperforming comparable efficient-transformer baselines in many settings. The approach supports deployment across a range of platforms and paves the way for extending tensor-reduction-based attention to additional vision tasks.

Abstract

Transformers have excelled in many tasks including vision. However, efficient deployment of transformer models in low-latency or high-throughput applications is hindered by the computation in the attention mechanism which involves expensive operations such as matrix multiplication and Softmax. To address this, we introduce ReduceFormer, a family of models optimized for efficiency with the spirit of attention. ReduceFormer leverages only simple operations such as reduction and element-wise multiplication, leading to greatly simplified architecture and improved inference performance, with up to 37% reduction in latency and 44% improvement in throughput, while maintaining competitive accuracy comparable to other recent methods. The proposed model family is suitable for edge devices where compute resource and memory bandwidth are limited, as well as for cloud computing where high throughput is sought after.

ReduceFormer: Attention with Tensor Reduction by Summation

TL;DR

This work addresses the bottleneck of self-attention in vision transformers by replacing matrix multiplications and Softmax with simple tensor reductions and element-wise operations. It introduces ReduceFormer, a family of models combining a Multi-Scale Local Context module with a ReduceFormer Attention mechanism that uses global summations to project and relate spatial features, yielding substantially reduced computational cost while preserving accuracy. Empirical results on ImageNet-1K show competitive performance with significant gains in latency and throughput on both edge devices (e.g., Drive Orin) and data-center GPUs (L40), outperforming comparable efficient-transformer baselines in many settings. The approach supports deployment across a range of platforms and paves the way for extending tensor-reduction-based attention to additional vision tasks.

Abstract

Transformers have excelled in many tasks including vision. However, efficient deployment of transformer models in low-latency or high-throughput applications is hindered by the computation in the attention mechanism which involves expensive operations such as matrix multiplication and Softmax. To address this, we introduce ReduceFormer, a family of models optimized for efficiency with the spirit of attention. ReduceFormer leverages only simple operations such as reduction and element-wise multiplication, leading to greatly simplified architecture and improved inference performance, with up to 37% reduction in latency and 44% improvement in throughput, while maintaining competitive accuracy comparable to other recent methods. The proposed model family is suitable for edge devices where compute resource and memory bandwidth are limited, as well as for cloud computing where high throughput is sought after.
Paper Structure (7 sections, 2 equations, 4 figures, 2 tables)

This paper contains 7 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Multi-Scale Local Context
  5. ReduceFormer Attention
  6. Experiments
  7. Conclusion

Figures (4)

  • Figure 1: The proposed architecture of ReduceFormer. Each stage can be composed with $L$ repetitions of MB Conv blocks sandler2018mobilenetv2 with/without ReduceFormer blocks.
  • Figure 2: A ReduceFormer block consists of two phases: local context learning and global context learning via ReduceFormer attention.
  • Figure 3: Multi-Scale Local Feature Encoding. Before upcoming block that performs attention in a global manner, this phase allows learning local contexts.
  • Figure 4: Details of ReduceFormer Attention. The output tensors are used in Eq. \ref{['eq:RF_eqn']} to compute attention.