Patches Are All You Need?

TL;DR

The paper investigates whether patch-based input representations, rather than Transformer attention, primarily drive modern vision-model performance. It introduces ConvMixer, a simple isotropic architecture that operates on patch embeddings using depthwise spatial mixing and pointwise channel mixing implemented entirely with standard convolutions. On ImageNet-1k, ConvMixer achieves competitive or superior accuracy to ViT/MLP-Mixer and remains close to DeiT/ResMLP within similar parameter budgets, suggesting the patch embedding strategy itself is a strong contributor. The work proposes ConvMixer as a strong, low-variance baseline for future patch-based architectures and highlights the potential of large-kernel convolutions in isotropic designs, with future work spanning larger patches, pretraining, and broader tasks.

Abstract

Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.

Figures (8)

• Figure 1: Accuracy vs. parameters, trained and evaluated on ImageNet-1k.
• Figure 2: ConvMixer uses "tensor layout" patch embeddings to preserve locality, and then applies $d$ copies of a simple fully-convolutional block consisting of large-kernel depthwise convolution followed by pointwise convolution, before finishing with global pooling and a simple linear classifier.
• Figure 3: Implementation of ConvMixer in PyTorch; see Appendix \ref{['apx-implementation']} for more implementations.
• Figure 4: Patch embedding weights for a ConvMixer-1024/20 with patch size 14 (see Table \ref{['models']}).
• Figure 5: Patch embedding weights for a ConvMixer-768/32 with patch size 7 (see Table \ref{['models']}).