ReScene4D: Temporally Consistent Semantic Instance Segmentation of Evolving Indoor 3D Scenes

TL;DR

This work formalizes temporally sparse 4D semantic instance segmentation (4DSIS) for evolving indoor scenes and introduces ReScene4D, a unified spatio-temporal framework that refines temporally shared instance queries without relying on dense observations. It jointly segments, identifies, and temporally associates instances across intermittently captured scans, aided by three information-sharing strategies and a new t-mAP metric that rewards temporal identity continuity. On 3RScan, ReScene4D achieves state-of-the-art performance, showcasing that cross-time information sharing improves both 4DSIS and conventional 3DSIS. The approach advances robust understanding of evolving indoor environments, though it underscores the need for larger, more dynamic 4D indoor datasets and further integration with recent 3DSIS advances.

Abstract

Indoor environments evolve as objects move, appear, or disappear. Capturing these dynamics requires maintaining temporally consistent instance identities across intermittently captured 3D scans, even when changes are unobserved. We introduce and formalize the task of temporally sparse 4D indoor semantic instance segmentation (SIS), which jointly segments, identifies, and temporally associates object instances. This setting poses a challenge for existing 3DSIS methods, which require a discrete matching step due to their lack of temporal reasoning, and for 4D LiDAR approaches, which perform poorly due to their reliance on high-frequency temporal measurements that are uncommon in the longer-horizon evolution of indoor environments. We propose ReScene4D, a novel method that adapts 3DSIS architectures for 4DSIS without needing dense observations. It explores strategies to share information across observations, demonstrating that this shared context not only enables consistent instance tracking but also improves standard 3DSIS quality. To evaluate this task, we define a new metric, t-mAP, that extends mAP to reward temporal identity consistency. ReScene4D achieves state-of-the-art performance on the 3RScan dataset, establishing a new benchmark for understanding evolving indoor scenes.

Figures (16)

• Figure 1: ReScene4D on 4D Semantic Instance Segmentation. Our method outperforms baselines which are unable to accurately assign instance identities of both static and changing objects, across multiple temporal observations of a scene (left). Our method (right) maintains instance identities between the observations, even when instances move or change.
• Figure 2: Overview of ReScene4D Architecture. Given a temporal sequence of $T$ 3D observations, hierarchical features for each temporal stage, preserving temporal distinction are extracted using a backbone encoder . A transformer-based query decoder iteratively refines spatio-temporal (ST) instance queries by jointly sampling across temporal hierarchical features . Given ST superpoint features and ST queries, the mask module predicts joint binary masks and semantic classes consistent across the sequence. Adaptations for 4DSIS are denoted in purple. Our temporal information sharing modules ①, ②, ③ facilitate cross-temporal consistency and shared learning via cross-time contrastive loss, ST mask pooling, and ST decoder serialization .
• Figure 3: Toy Examples for Temporal Metrics. Right: summary table showing IoU and t-IoU scores for four cases.
• Figure 4: Temporal Semantic Instance Segmentation Comparison.
• Figure 5: Temporal Semantic Instance Segmentation Comparison---Close up of Figure \ref{['fig:main_example']} with baseline comparison. While Mask4Former has a tendency to merge multiple objects into one instance and Mask3D fails to track identities across time, ReScene4D consistently identifies and tracks object instances across temporal scans, be that dynamic or static. It even considers the two bookcases next to the curtains as separate instances, in contrast to ground truth annotations. Non-systematic annotation of small objects throughout the dataset leads to our method missing some of the pillows on the couch, as well as the identification of a box on the top shelf of the bookcase as a separate instance.