Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

Xiawei Li, Qingyuan Xu, Jing Zhang, Tianyi Zhang, Qian Yu, Lu Sheng, Dong Xu

TL;DR

This work tackles weakly supervised 3D semantic segmentation with scene-level labels by exploiting RGB-D multimodality through a Multi-modality Affinity (MMA) module. It introduces a two-stream architecture (RGB-appended and RGB-masked geometry), a multi-scale affinity computation, and long-tail normalization of classifier weights to robustly infer point affinities and refine MIL-based learning. The method achieves substantial improvements on ScanNet and S3DIS, outperforming state-of-the-art scene-level WSSS methods by several percent mIoU and showing strong per-class gains, especially for geometry- and color-distinguishable categories. By enabling cross-modal, multi-scale affinity propagation and cross-stream consistency, MMA enhances pseudo-label quality and self-training effectiveness, offering a practical path to high-quality 3D segmentation with weak supervision.

Abstract

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.

Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

TL;DR

This work tackles weakly supervised 3D semantic segmentation with scene-level labels by exploiting RGB-D multimodality through a Multi-modality Affinity (MMA) module. It introduces a two-stream architecture (RGB-appended and RGB-masked geometry), a multi-scale affinity computation, and long-tail normalization of classifier weights to robustly infer point affinities and refine MIL-based learning. The method achieves substantial improvements on ScanNet and S3DIS, outperforming state-of-the-art scene-level WSSS methods by several percent mIoU and showing strong per-class gains, especially for geometry- and color-distinguishable categories. By enabling cross-modal, multi-scale affinity propagation and cross-stream consistency, MMA enhances pseudo-label quality and self-training effectiveness, offering a practical path to high-quality 3D segmentation with weak supervision.

Abstract

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.
Paper Structure (29 sections, 8 equations, 3 figures, 6 tables)

This paper contains 29 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Weakly-supervised Point Cloud Segmentation with Sparse Labels.
  4. Weakly-supervised Point Cloud Segmentation with Scene-level Labels.
  5. Methodology
  6. Overview.
  7. MIL-based 3D Semantic Segmentation
  8. Feature Extraction Module.
  9. Segmentation Module.
  10. Loss Function.
  11. Long-tail-aware Multi-modality Point Affinity
  12. Multi-modality Point Affinity.
  13. Long-tail Aware Affinity Enhancement.
  14. Objective Functions
  15. Experiments
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Pipeline of the proposed method. A two-stream architecture with shared parameters is adopted, where the two streams take pure geometric point clouds and RGB-appended point clouds as inputs, respectively. Three main modules are involved: a feature extraction module, a segmentation module, and a multi-modality affinity inference module.
  • Figure 2: Qualitative segmentation results: (a) Point cloud data, (b) Ground-truth segmentation results, (c) Baseline results, (d) Results of "Baseline+Multi-modality affinity w/o $\hat{W}$" variant of our method, (e) Results of our final method ("Baseline+Multi-modality affinity").
  • Figure 3: Class relationship maps enhanced by point affinity.