Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Segment Any 3D Thing as Instance Tracking

Hanshi Wang, Zijian Cai, Jin Gao, Yiwei Zhang, Weiming Hu, Ke Wang, Zhipeng Zhang

TL;DR

This work reframes online 3D instance segmentation as an instance-tracking problem to address temporal coherence and fragmentation. It introduces a lightweight tracking architecture with Long-Term Memory, Short-Term Memory, and Spatial Consistency Learning to propagate identities, inject immediate contextual updates, and merge fragmented masks, achieving state-of-the-art AP on ScanNet200 while sustaining real-time throughput. The approach demonstrates strong generalization across ScanNet, SceneNN, and 3RScan and provides extensive ablations that validate the contribution of each module. The work advances embodied perception by enabling robust, temporally-consistent 3D segmentation under partial views and open-vocabulary scenarios.

Abstract

Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets.

Online Segment Any 3D Thing as Instance Tracking

TL;DR

This work reframes online 3D instance segmentation as an instance-tracking problem to address temporal coherence and fragmentation. It introduces a lightweight tracking architecture with Long-Term Memory, Short-Term Memory, and Spatial Consistency Learning to propagate identities, inject immediate contextual updates, and merge fragmented masks, achieving state-of-the-art AP on ScanNet200 while sustaining real-time throughput. The approach demonstrates strong generalization across ScanNet, SceneNN, and 3RScan and provides extensive ablations that validate the contribution of each module. The work advances embodied perception by enabling robust, temporally-consistent 3D segmentation under partial views and open-vocabulary scenarios.

Abstract

Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets.

Paper Structure

This paper contains 13 sections, 14 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overall Architecture
  5. Long-Term Memory for Instance Association
  6. Short-Term Memory for Instance Update
  7. Spatial Consistency Learning for Robust Association
  8. Loss Functions
  9. Experiments
  10. Experiment Settings
  11. Comparison with State-of-the-arts
  12. Ablation Studies and Further Analysis
  13. Conclusion and Limitation

Figures (2)

  • Figure 1: This diagram delineates the operational mechanisms of our constituent modules. Spatial Consistency Learning (SCL) mitigates the over-segmentation tendencies of VFM by employing a one-to-many supervision strategy during the training phase and utilizing learning-based mask integration at the inference stage. The Short-term Memory (STM) module enriches current instance representations by integrating observational data from prior frames. Furthermore, the Long-term Memory (LTM) module is engineered to associate instances, segmented by the Visual Front-end Module (VFM), with established tracklets in memory, consequently enhancing temporal consistency.
  • Figure 2: Visualization of segmentation results on ScanNet200 dataset.