Minuet: Accelerating 3D Sparse Convolutions on GPUs

TL;DR

Minuet tackles inefficiencies in GPU-based Sparse Convolution for 3D point clouds by exchanging hash-table kernel-map construction for a memory-efficient segmented sorting and double-traversed binary search, and by making Gather/Scatter more adaptive through autotuned tiling and padding-aware GEMM grouping. The four core contributions are: (1) Segmented Query Sorting for Map step to exploit on-chip memory and cache locality, (2) Double-Traversed Binary Search to reduce comparisons while maintaining cache efficiency, (3) Autotuned Gather/Scatter tile sizes to adapt to layer, dataset, and hardware, and (4) Padding-Efficient GEMM Grouping to minimize zero-padding and kernel-launch overheads. Across diverse networks, datasets, and GPUs, Minuet delivers substantial gains, including a $1.74\times$ average end-to-end speedup (up to $2.22\times$) and a notable $19.2\times$ map-step speedup (up to $26.8\times$) over hash-table-based baselines, with consistent improvements in GMaS as well. The work significantly advances practical sparse 3D inference by optimizing memory access, data movement, and compute patterns, and is publicly available as open source.

Abstract

Sparse Convolution (SC) is widely used for processing 3D point clouds that are inherently sparse. Different from dense convolution, SC preserves the sparsity of the input point cloud by only allowing outputs to specific locations. To efficiently compute SC, prior SC engines first use hash tables to build a kernel map that stores the necessary General Matrix Multiplication (GEMM) operations to be executed (Map step), and then use a Gather-GEMM-Scatter process to execute these GEMM operations (GMaS step). In this work, we analyze the shortcomings of prior state-of-the-art SC engines, and propose Minuet, a novel memory-efficient SC engine tailored for modern GPUs. Minuet proposes to (i) replace the hash tables used in the Map step with a novel segmented sorting double-traversed binary search algorithm that highly utilizes the on-chip memory hierarchy of GPUs, (ii) use a lightweight scheme to autotune the tile size in the Gather and Scatter operations of the GMaS step, such that to adapt the execution to the particular characteristics of each SC layer, dataset, and GPU architecture, and (iii) employ a padding-efficient GEMM grouping approach that reduces both memory padding and kernel launching overheads. Our evaluations show that Minuet significantly outperforms prior SC engines by on average $1.74\times$ (up to $2.22\times$) for end-to-end point cloud network executions. Our novel segmented sorting double-traversed binary search algorithm achieves superior speedups by $15.8\times$ on average (up to $26.8\times$) over prior SC engines in the Map step. The source code of Minuet is publicly available at https://github.com/UofT-EcoSystem/Minuet.

Figures (20)

• Figure 1: Dense convolution versus sparse convolution.
• Figure 2: The SC execution can be broken down into two steps. For simplicity, we use 2D coordinates for illustration.
• Figure 3: L2 cache hit ratio in building kernel maps of the Map step on RTX 3090 for various kernel map building implementations.
• Figure 4: The performance of a Gather operation, when varying the (a) input channel size, (b) real dataset, and (c) GPU architecture.
• Figure 5: Various approaches to execute GEMM operations in SC, where one blue and white squares denote one actual input feature vector and one zero-padded feature vector, respectively. Assuming $x$ and $y$ are the number of padded feature vectors and actual input feature vectors, respectively, the padding overhead is defined as $(x / y)$.