Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

TL;DR

This work revisits the underexplored use of large convolutional kernels in CNNs as a path to close the gap with Vision Transformers (ViTs). It introduces RepLKNet, a pure CNN that employs very large depthwise convolutions up to $31\times31$ to enlarge the effective receptive field, guided by five practical design principles and a structural re-parameterization strategy. Across ImageNet, semantic segmentation, and object detection, RepLKNet delivers competitive or superior performance with favorable latency, and demonstrates stronger shape bias and downstream transfer than traditional small-kernel CNNs. The findings suggest kernel size as a scalable design dimension for CNNs and offer concrete guidelines and architectures to exploit large kernels effectively in practical vision tasks.

Abstract

We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depth-wise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31x31, in contrast to commonly used 3x3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.

Figures (6)

• Figure 1: The Effective Receptive Field (ERF) of ResNet-101/152 and RepLKNet-13/31 respectively. A more widely distributed dark area indicates a larger ERF. More layers (e.g., from ResNet-101 to ResNet-152) help little in enlarging ERFs. Instead, our large kernel model RepLKNet effectively obtains large ERFs.
• Figure 2: An example of re-parameterizing a small kernel (e.g., 3$\times$3) into a large one (e.g., 7$\times$7). See ding2019acnetding2021repvgg for details.
• Figure 3: Illustration to convolution with small feature map and large kernel. Two outputs at adjacent locations only share a part of kernel weights. Translational equivariance does not strictly hold.
• Figure 4: RepLKNet comprises Stem, Stages and Transitions. Except for depth-wise (DW) large kernel, the other components include DW 3$\times$3, dense 1$\times$1 conv, and batch normalization ioffe2015batch (BN). Note that every conv layer has a following BN, which are not depicted. Such conv-BN sequences use ReLU as the activation function, except those before the shortcut-addition (as a common practice he2016deepmbv2) and those preceding GELU hendrycks2016gaussian.
• Figure 5: Shape bias of RepLKNet, Swin, and ResNet-152 pretrained on ImageNet-1K or 22K. The scatters represent the shape bias of 16 categories, and the vertical lines are the averages across categories (note RepLKNet-3 and ResNet-152 are very close).