A Unified Framework for 3D Scene Understanding

TL;DR

UniSeg3D is a unified 3D scene understanding framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary segmentation tasks within a single model and facilitates inter-task knowledge sharing, thereby promoting comprehensive 3D scene understanding.

Abstract

We propose UniSeg3D, a unified 3D scene understanding framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary segmentation tasks within a single model. Most previous 3D segmentation approaches are typically tailored to a specific task, limiting their understanding of 3D scenes to a task-specific perspective. In contrast, the proposed method unifies six tasks into unified representations processed by the same Transformer. It facilitates inter-task knowledge sharing, thereby promoting comprehensive 3D scene understanding. To take advantage of multi-task unification, we enhance performance by establishing explicit inter-task associations. Specifically, we design knowledge distillation and contrastive learning methods to transfer task-specific knowledge across different tasks. Experiments on three benchmarks, including ScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3D consistently outperforms current SOTA methods, even those specialized for individual tasks. We hope UniSeg3D can serve as a solid unified baseline and inspire future work. Code and models are available at https://github.com/dk-liang/UniSeg3D.

Figures (9)

• Figure 1: Comparisons between the proposed method and current SOTA approaches specialized for specific tasks. (a) Representative specialized approaches on six tasks. (b) OneFormer3D, a recent unified framework, achieves SOTA performance on three generic segmentation tasks in one inference. (c) The proposed unified framework achieves six tasks in one inference. (d) Our method outperforms current SOTA approaches across six tasks involving two modalities using a single model.
• Figure 2: The framework of UniSeg3D. This is a simple framework handling six tasks in parallel without any modules specialized for specific tasks. We take advantage of multi-task unification and enhance the performance through building associations between the supported tasks. Specifically, knowledge distillation transfers insights from interactive segmentation to the other tasks, while contrastive learning establishes connections between interactive segmentation and referring segmentation.
• Figure 3: Illustration of the inter-task association. (a) A challenging case requiring the distinction of textual positional information within the expressions. (b) A contrastive learning matrix for vision-text pairs, where a ranking rule is employed to suppress incorrect pairings. (c) Knowledge distillation across multi-task predictions.
• Figure I: Comparison with existing instance segmentation methods on ScanNet20. UniSeg3D achieves highly competitive performance.
• Figure IV: Visualization of segmentation results obtained by UniSeg3D on ScanNet20 validation split.