More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

TL;DR

The paper tackles the challenge of scaling convolutional neural networks to extremely large kernels beyond 31×31. It introduces SLaK, a pure CNN built on a two-step recipe: decompose large kernels into two parallel non-square paths and employ dynamic sparsity to expand width, achieving $51$×$51$ kernels with efficient scaling. Across ImageNet-1K, ADE20K, PASCAL VOC 2007, and COCO, SLaK matches or surpasses state-of-the-art CNNs and Transformer-based models, aided by analyses of effective receptive fields and kernel-scaling efficiency. The work demonstrates that sparsity, paired with kernel decomposition, enables large-kernel CNNs to outperform prior approaches and motivates further hardware support for sparse computation. This approach provides a practical path to leverage extremely large receptive fields in vision models without resorting to attention-based architectures.

Abstract

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO.

Figures (6)

• Figure 1: Large depth-wise kernel (e.g., 51$\times$51) paradigms of ConvNeXt, RepLKNet, and SLaK. Dark blue squares refer to the dense weights in convolutional kernels. Light blue squares refer to the sparse weights in convolutional kernels.
• Figure 2: Dynamic sparsity. Dynamic sparsity allows us to construct and train initially sparse neural networks (sparse kernels) from scratch. During training, it dynamically adjusts the sparse weights by pruning the least important weights and adding new. Such dynamic procedure gradually optimizes the sparse kernels to a good pattern and hence encourages a more elaborate capture of local features.
• Figure 3: Effective receptive field (ERF) of models with various kernel sizes. SLaK is not only able to capture long-range dependence but also the local context features.
• Figure 4: Left:Scaling efficiency. The number of GFLOPs (left) and parameters (right) as the kernel size in ConvNeXt-T scales up. Right:Real inference time latency (ms). Real inference time latency of depth-size convolutions with different kernel size. The results are obtained on a single A100 GPU with PyTorch 1.10.0 + cuDNN 8.2.0, in FP32 precision, using no special hardware accelerator.
• Figure 5: Left: Effect of adaptation rate $p$ on the performance of SLaK-T. $f$ is set as 100. Right: Effect of the adaptation frequency $f$ on the performance of SLaK-T. $p$ is set as 0.3.