Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

iFormer: Integrating ConvNet and Transformer for Mobile Application

Chuanyang Zheng

TL;DR

iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention to create a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications.

Abstract

We present a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications. iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention. The local interactions are derived from transforming a standard convolutional network, \textit{i.e.}, ConvNeXt, to design a more lightweight mobile network. Our newly introduced mobile modulation attention removes memory-intensive operations in MHA and employs an efficient modulation mechanism to boost dynamic global representational capacity. We conduct comprehensive experiments demonstrating that iFormer outperforms existing lightweight networks across various tasks. Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k with a latency of only 1.10 ms on an iPhone 13, surpassing the recently proposed MobileNetV4 under similar latency constraints. Additionally, our method shows significant improvements in downstream tasks, including COCO object detection, instance segmentation, and ADE20k semantic segmentation, while still maintaining low latency on mobile devices for high-resolution inputs in these scenarios.

iFormer: Integrating ConvNet and Transformer for Mobile Application

TL;DR

iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention to create a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications.

Abstract

We present a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications. iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention. The local interactions are derived from transforming a standard convolutional network, \textit{i.e.}, ConvNeXt, to design a more lightweight mobile network. Our newly introduced mobile modulation attention removes memory-intensive operations in MHA and employs an efficient modulation mechanism to boost dynamic global representational capacity. We conduct comprehensive experiments demonstrating that iFormer outperforms existing lightweight networks across various tasks. Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k with a latency of only 1.10 ms on an iPhone 13, surpassing the recently proposed MobileNetV4 under similar latency constraints. Additionally, our method shows significant improvements in downstream tasks, including COCO object detection, instance segmentation, and ADE20k semantic segmentation, while still maintaining low latency on mobile devices for high-resolution inputs in these scenarios.
Paper Structure (42 sections, 7 equations, 5 figures, 18 tables)

This paper contains 42 sections, 7 equations, 5 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficient Convolutional Networks
  4. Efficient Vision Transformers
  5. Method
  6. Preparing ConvNeXt
  7. Lightening Baseline
  8. Seeing Better with Early Convolutions
  9. Normalization
  10. Going Deeper
  11. Stage Ratio
  12. Kernel Size
  13. Single-Head Modulation Attention
  14. Single-Head vs. Multi-Head
  15. Modulation Attention
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Comparison of latency and accuracy between our iFormer and other existing methods on ImageNet-1k. The latency is measured on an iPhone 13. Our iFormer is Pareto-optimal.
  • Figure 2: Illustration of the evolution from the ConvNeXt baseline towards the lightweight iFormer. The orange bars are model accuracies and the light blue bars are model latencies. We also include a red latency outline for better visualization.
  • Figure 3: The distribution of average cosine similarity among multiple heads within the MHA mechanism. As the layer depth increases, the similarity goes higher.
  • Figure 4: Overview of iFormer architecture, detailed convolutional stem, block design, and SHMA. The hatched area in SHMA indicates extra memory-intensive reshaping operations that are eliminated by SHMA. $S(\cdot)$ denotes the softmax function. $R$ is the ratio for reducing channels of query and key. It is set to 2 in iFormer. We omit BN following project or convolution for simplicity.
  • Figure 5: Comparison of SHMA and SHA in SHViT. In SHViT, $rC$ channels are utilized for spatial attention, where $r$ is set to $\frac{1}{4.67}$. SHMA projects the input into a higher dimension of $\frac{1}{2}$C (i.e., R=2) and avoids split and concatenation operations.