MetaFormer Baselines for Vision

TL;DR

This work probes the capacity of MetaFormer, the abstracted Transformer architecture for vision, by focusing on basic token mixers rather than novel designs. It introduces IdentityFormer and RandFormer as lower-bound and universal baselines, and ConvFormer and CAFormer as strong baselines using conventional token mixers, with CAFormer achieving a new ImageNet-1K record of 85.5% top-1 accuracy at 224^2 and improving further with ImageNet-21K pretraining. The study also introduces StarReLU, a cheaper activation that reduces FLOPs by up to 71% while delivering accuracy gains, and shows that scaling refinements like ResScale further enhance performance. Across ImageNet-1K/21K, COCO, and ADE20K, these MetaFormer baselines demonstrate competitive or superior accuracy and robustness compared to state-of-the-art models, while maintaining favorable throughput. Together, they establish MetaFormer baselines as strong, dependable references for vision model development and highlight the practicality of simple token mixers combined with efficient activations.

Abstract

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, without focusing on token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves >80% accuracy on ImageNet-1K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of >81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1K: it achieves an accuracy of 85.5% at 224x224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with GELU yet achieves better performance. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

Figures (4)

• Figure 1: Performance of MetaFormer baselines and other state-of-the-art models on ImageNet-1K at $224^2$ resolution. The architectures of our proposed models are shown in Figure \ref{['fig:overall_framework']}. (a) IdentityFormer/RandFormer achieve over 80%/81% accuracy, indicating MetaFormer has solid lower bound of performance and works well on arbitrary token mixers. The accuracy of well-trained ResNet-50 resnet is from rsb. (b) Without novel token mixers, pure CNN-based ConvFormer outperforms ConvNeXt convnext, while CAFormer sets a new record of 85.5% accuracy on ImageNet-1K at $224^2$ resolution under normal supervised training without external data or distillation.
• Figure 2: (a-d) Overall frameworks of IdentityFormer, RandFormer, ConvFormer and CAFormer. Similar to resnetpvtswin, the models adopt hierarchical architecture of 4 stages, and stage $i$ has $L_i$ blocks with feature dimension $D_i$. Each downsampling module is implemented by a layer of convolution. The first downsampling has kernel size of 7 and stride of 4, while the last three ones have kernel size of 3 and stride of 2. (e-h) Architectures of IdentityFormer, RandFormer, ConvFormer and Transformer blocks, which have token mixer of identity mapping, global random mixing (Equation \ref{['eqn:random_mixing']}), separable depthwise convolutions chollet2017xceptionmamalet2012simplifyingmobilenetv2 (Equation \ref{['eqn:convolutions']}) or vanilla self-attention transformer, respectively.
• Figure 3: ImageNet-1K validation accuracy vs. MACs/Model Size at the resolution of $224^2$. Models with token (feature) mixing based on convolution, attention or hybrid are presented by $\square$, $\triangle$ or $\mathord{ { \hbox{$\m@th\bigcirc$} } }$ respectively.
• Figure 4: Trade-off between accuracy and inference throughput. The throughputs are measured on an NVIDIA A100 GPU with batch size of 128 and TF32.