LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernels

TL;DR

An efficient and effective Large Sparse Kernel 3D Neural Network (LSK3DNet) that leverages dynamic pruning to amplify the 3D kernel size and significantly reduces model size and computational cost is proposed.

Abstract

Autonomous systems need to process large-scale, sparse, and irregular point clouds with limited compute resources. Consequently, it is essential to develop LiDAR perception methods that are both efficient and effective. Although naively enlarging 3D kernel size can enhance performance, it will also lead to a cubically-increasing overhead. Therefore, it is crucial to develop streamlined 3D large kernel designs that eliminate redundant weights and work effectively with larger kernels. In this paper, we propose an efficient and effective Large Sparse Kernel 3D Neural Network (LSK3DNet) that leverages dynamic pruning to amplify the 3D kernel size. Our method comprises two core components: Spatial-wise Dynamic Sparsity (SDS) and Channel-wise Weight Selection (CWS). SDS dynamically prunes and regrows volumetric weights from the beginning to learn a large sparse 3D kernel. It not only boosts performance but also significantly reduces model size and computational cost. Moreover, CWS selects the most important channels for 3D convolution during training and subsequently prunes the redundant channels to accelerate inference for 3D vision tasks. We demonstrate the effectiveness of LSK3DNet on three benchmark datasets and five tracks compared with classical models and large kernel designs. Notably, LSK3DNet achieves the state-of-the-art performance on SemanticKITTI (i.e., 75.6% on single-scan and 63.4% on multi-scan), with roughly 40% model size reduction and 60% computing operations reduction compared to the naive large 3D kernel model.

Figures (8)

• Figure 1: Illustrations on Spatial-wise Group Convolutionchen2022scaling, SDS, and CWS. The spatial dimensions $K_s$ (i.e., $K_1$, $K_2$, $K_3$) and channel dimensions ($D_{in}$/$D_{out}$) are shown. Spatial-wise Group Convolution shares the weights within each spatial group during training, leading to redundant model weights. In contrast, SDS removes non-salient weights and redundancies that are not sensitive to the input in each group, while CWS eliminates redundant weights in a channel-wise manner (Sec. \ref{['sec:introduction']}).
• Figure 2: $_{\!}$Performance$_{\!}$ (mIoU)$_{\!}$vs.$_{\!}$ Inference$_{\!}$ Speed$_{\!}$ (FPS)$_{\!}$ on$_{\!}$ SemanticKITTI behley2019semantickitti single-scan challenge (Sec. \ref{['sec:introduction']}).
• Figure 3: Spatial-wise Dynamic Sparsity. The utilization of SDS enables us to create and train sparse kernel 3D neural networks from the beginning. The sparse weights in each spatial group are firstly initialized by Sparse Kernel Initialization (Eq.$_{\!}$ (\ref{['eq:Initialization']})), and then regularly altered by discarding the least significant connections and introducing new ones (Eq.$_{\!}$ (\ref{['eq:pruning']})). The sparse kernels are steadily improved by this dynamic process, which leads to a more thorough collection of local features. Note that different spatial sparse group has different sparse distribution. Here we take a sparsity of 22% for example (Sec. \ref{['sec:Sparsity']} and Alg. \ref{['alg:algorithm1']}).
• Figure 4: LSK3DNet and LSK Block. (1)$_{\!}$ LSK3DNet: Point clouds are fed into voxelizer for voxel-wise features. Then we extract features with LSK Block and Point Branch. The final prediction is a point-wise output. Here MLPs is the standard Multi-Layer Perceptrons.$_{\!}$ (2)$_{\!}$ Large Sparse Kernel Block (LSK Block) (Sec. \ref{['sec:Objective']}).
• Figure 5: Effective Receptive Fields (ERFs) of Baseline and LSK- 3DNet. LSK3DNet has a larger ERF size. Additionally, we provide visualization of learned sparse kernels, where all weight values have taken the absolute value and normalization operations. The black areas indicate positions with weight values of zero.