RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone

TL;DR

RapidNet presents a purely CNN mobile backbone built on Multi-Level Dilated Convolutions to enlarge receptive fields and enable multi-scale feature interactions without attention. By integrating MLDC with a large-kernel FFN and reparameterizable depthwise convolutions, RapidNet achieves competitive or superior accuracy and latency versus state-of-the-art mobile CNNs, ViTs, ViGs, and hybrids across ImageNet-1K, COCO, and ADE20K. The approach yields explicit gains in top-1 accuracy (e.g., RapidNet-Ti 76.3% on ImageNet-1K with 0.9 ms on iPhone), AP on object detection/segmentation, and mIoU on segmentation, demonstrating that well-designed CNNs can outperform more complex hybrid models on mobile hardware. These results highlight MLDC as a practical mechanism to trade increased receptive field for speed, offering a path toward more efficient mobile vision systems with strong cross-task transfer.

Abstract

Vision transformers (ViTs) have dominated computer vision in recent years. However, ViTs are computationally expensive and not well suited for mobile devices; this led to the prevalence of convolutional neural network (CNN) and ViT-based hybrid models for mobile vision applications. Recently, Vision GNN (ViG) and CNN hybrid models have also been proposed for mobile vision tasks. However, all of these methods remain slower compared to pure CNN-based models. In this work, we propose Multi-Level Dilated Convolutions to devise a purely CNN-based mobile backbone. Using Multi-Level Dilated Convolutions allows for a larger theoretical receptive field than standard convolutions. Different levels of dilation also allow for interactions between the short-range and long-range features in an image. Experiments show that our proposed model outperforms state-of-the-art (SOTA) mobile CNN, ViT, ViG, and hybrid architectures in terms of accuracy and/or speed on image classification, object detection, instance segmentation, and semantic segmentation. Our fastest model, RapidNet-Ti, achieves 76.3\% top-1 accuracy on ImageNet-1K with 0.9 ms inference latency on an iPhone 13 mini NPU, which is faster and more accurate than MobileNetV2x1.4 (74.7\% top-1 with 1.0 ms latency). Our work shows that pure CNN architectures can beat SOTA hybrid and ViT models in terms of accuracy and speed when designed properly.

Figures (4)

• Figure 1: Comparison of accuracy on ImageNet-1K, Average Precision (AP) on MS COCO, and mean Intersection over Union (mIoU) on ADE20K. a) RapidNet achieves the best accuracy-GMACs tradeoff on all model sizes compared. b) RapidNet achieves the best AP-GMACs tradeoff on all model sizes compared. c) RapidNet achieves the best mIoU-GMACs tradeoff on all model sizes compared. GMACs are computed using a resolution of 224 $\times$ 224 for a) and a resolution of 512 $\times$ 512 for b) and c).
• Figure 2: Comparison of regular and dilated convolution. a) In a regular convolution with a kernel size of 3 in a 7 $\times$ 7 image, we can see the convolution is applied to the 3 $\times$ 3 patches in the grid. b) In a dilated convolution with a kernel size of 3 and a dilation factor of 2 in a 7 $\times$ 7 image, we can see the convolution is applied to the 5 $\times$ 5 patches in the grid thereby expanding the receptive field. This is done by skipping the patches in red in the convolution, preserving the parameters needed for 3 $\times$ 3 convolution, but expanding the receptive field to that of a 5 $\times$ 5 convolution.
• Figure 3: RapidNet architecture. (a) Network architecture showing the stages and layers, where N1i, N2i, N3i, N4i, N3d, and N4d represent the number of Inverted Residual Blocks and Dilated Convolution Blocks in the RapidNet-Ti, S, M, and B configurations and S represents the stride of the convolutions. (b) The Conv Stem. (c) Inverted Residual Block. (d) Downsample. (e) Dilated Convolution Block. (f) Multi-Level Dilated Convolution (MLDC) Block. (g) Large Kernel FFN.
• Figure 4: Comparison of accuracy versus latency on ImageNet-1K. RapidNet achieves the best accuracy-latency tradeoff on all model sizes compared.