ParFormer: A Vision Transformer with Parallel Mixer and Sparse Channel Attention Patch Embedding

TL;DR

Results validate ParFormer as a highly efficient and scalable model for both high-performance and resource-constrained scenarios, making it an ideal solution for edge-based AI applications.

Abstract

Convolutional Neural Networks (CNNs) and Transformers have achieved remarkable success in computer vision tasks. However, their deep architectures often lead to high computational redundancy, making them less suitable for resource-constrained environments, such as edge devices. This paper introduces ParFormer, a novel vision transformer that addresses this challenge by incorporating a Parallel Mixer and a Sparse Channel Attention Patch Embedding (SCAPE). By combining convolutional and attention mechanisms, ParFormer improves feature extraction. This makes spatial feature extraction more efficient and cuts down on unnecessary computation. The SCAPE module further reduces computational redundancy while preserving essential feature information during down-sampling. Experimental results on the ImageNet-1K dataset show that ParFormer-T achieves 78.9\% Top-1 accuracy with a high throughput on a GPU that outperforms other small models with 2.56$\times$ higher throughput than MobileViT-S, 0.24\% faster than FasterNet-T2, and 1.79$\times$ higher than EdgeNeXt-S. For edge device deployment, ParFormer-T excels with a throughput of 278.1 images/sec, which is 1.38 $\times$ higher than EdgeNeXt-S and 2.36$\times$ higher than MobileViT-S, making it highly suitable for real-time applications in resource-constrained settings. The larger variant, ParFormer-L, reaches 83.5\% Top-1 accuracy, offering a balanced trade-off between accuracy and efficiency, surpassing many state-of-the-art models. In COCO object detection, ParFormer-M achieves 40.7 AP for object detection and 37.6 AP for instance segmentation, surpassing models like ResNet-50, PVT-S and PoolFormer-S24 with significantly higher efficiency. These results validate ParFormer as a highly efficient and scalable model for both high-performance and resource-constrained scenarios, making it an ideal solution for edge-based AI applications.

Figures (3)

• Figure 1: The overall ParFormer Architecture, ParFormer used pyramid architecture with 4 stages. The feature spatial dimension is decreased from stride 4 to stride 32 using SCAPE module with increased channel dimension in each stage are $[C_1, C_2, C_3, C_4]$. In each stage, the Parallel Mixer is used with different ratio configurations.
• Figure 2: Sparse Channel Attention Patch Embedding. The input feature can be the image or feature maps from the previous stage that downsized using stride convolution. Non-linearity function after linear layer in the SCAM module is Sigmoid function. Finally the element wise product is used to re-calibrated the downsized feature maps.
• Figure 3: Qualitative results of object detection and instance segmentation on COCO test2017lin2014microsoft