UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition

TL;DR

Large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient, and the proposed model achieves state- of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture.

Abstract

Large-kernel convolutional neural networks (ConvNets) have recently received extensive research attention, but two unresolved and critical issues demand further investigation. 1) The architectures of existing large-kernel ConvNets largely follow the design principles of conventional ConvNets or transformers, while the architectural design for large-kernel ConvNets remains under-addressed. 2) As transformers have dominated multiple modalities, it remains to be investigated whether ConvNets also have a strong universal perception ability in domains beyond vision. In this paper, we contribute from two aspects. 1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep. Following such guidelines, our proposed large-kernel ConvNet shows leading performance in image recognition (ImageNet accuracy of 88.0%, ADE20K mIoU of 55.6%, and COCO box AP of 56.4%), demonstrating better performance and higher speed than the recent powerful competitors. 2) We discover large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient. With certain modality-related preprocessing approaches, the proposed model achieves state-of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture. All the code and models are publicly available on GitHub and Huggingface.

Figures (7)

• Figure 1: Architectural design of UniRepLKNet. A LarK Block comprises a Dilated Reparam Block proposed in this paper, an SE Block hu2018squeeze, an FFN, and Batch Normalization (BN) ioffe2015batch layers. The only difference between a SmaK Block and a LarK Block is that the former uses a depth-wise 3$\times$3 conv layer in replacement of the Dilated Reparam Block in the latter. Stages are connected by downsampling blocks implemented by stride-2 dense 3$\times$3 conv layers. We may flexibly arrange the blocks in different stages and the details of our provided instances are shown in Table \ref{['table-instances']}.
• Figure 2: Dilated Reparam Block uses dilated small-kernel conv layers to enhance a non-dilated large-kernel layer. Such dilated layers are equivalent to a non-dilated conv layer with a larger sparse kernel, as shown from the parameter perspective, so that the whole block can be equivalently transformed into a single large-kernel conv. This example shows $K$=9, and we may use more dilated layers for larger $K$.
• Figure 3: Options of the extra structures to increase the depth.
• Figure 4: Pytorch code to convert a dilated conv layer's small kernel to a non-dilated layer's larger sparse kernel.
• Figure 5: A test case demonstrating the equivalency of the transformation.