Small but Mighty: Enhancing 3D Point Clouds Semantic Segmentation with U-Next Framework

TL;DR

Small but Mighty: Enhancing 3D Point Clouds Semantic Segmentation with U-Next Framework introduces U-Next to address the semantic gap in multi-scale 3D point cloud segmentation. The method stacks multiple U-Net $L^1$ codecs with cross-scale fusion and multi-level deep supervision to learn fine-grained representations. Empirical results on S3DIS, Toronto3D, and SensatUrban show consistent improvements over standard backbones like RandLA-Net, with mIoU gains ranging from $2.2\%$ to $10.1\%$ and robust gains across tasks. The framework is lightweight, generalizable to other backbones, and demonstrates potential as a plug-in for future 3D segmentation research.

Abstract

We study the problem of semantic segmentation of large-scale 3D point clouds. In recent years, significant research efforts have been directed toward local feature aggregation, improved loss functions and sampling strategies. While the fundamental framework of point cloud semantic segmentation has been largely overlooked, with most existing approaches rely on the U-Net architecture by default. In this paper, we propose U-Next, a small but mighty framework designed for point cloud semantic segmentation. The key to this framework is to learn multi-scale hierarchical representations from semantically similar feature maps. Specifically, we build our U-Next by stacking multiple U-Net $L^1$ codecs in a nested and densely arranged manner to minimize the semantic gap, while simultaneously fusing the feature maps across scales to effectively recover the fine-grained details. We also devised a multi-level deep supervision mechanism to further smooth gradient propagation and facilitate network optimization. Extensive experiments conducted on three large-scale benchmarks including S3DIS, Toronto3D, and SensatUrban demonstrate the superiority and the effectiveness of the proposed U-Next architecture. Our U-Next architecture shows consistent and visible performance improvements across different tasks and baseline models, indicating its great potential to serve as a general framework for future research.

Figures (11)

• Figure 1: Performance of various algorithms on S3DIS dataset (Area 5) with different frameworks. Our U-Next framework demonstrated a notable advantage over both U-Net and U-Net++.
• Figure 2: Illustration of the proposed U-Next and U-Net $L^1$. (A) The detailed architecture of the proposed U-Next. (B) The U-Net $L^1$ sub-network with deep supervision. FC: Fully Connected layer; DP: Dropout; Deep Sup.: Deep Supervision.
• Figure 3: Qualitative comparisons of RandLA-Net with different architectures on S3DIS (Area 5). MDS: Multi-level deep supervision.
• Figure 4: Detailed level evolution of U-Net variants.
• Figure 5: Illustration of irregular 3D point cloud processing and regular 2D image processing. RS: random sampling; MP: max pooling. Typically, max pooling is used to down-sample regular 2D images, in this process, structured CNNs are used to capture local feature maps; random sampling or farthest point sampling is used to down-sample irregular 3D point clouds, in this process, unstructured nearest neighbor aggregation are used to capture local feature maps. Then, structured transposed CNNs and pixel shuffle are usually used for 2D images to recover high-resolution information, whereas uneven nearest neighbor interpolations are usually used for 3D point clouds.