Consistent Instance Field for Dynamic Scene Understanding

Junyi Wu; Van Nguyen Nguyen; Benjamin Planche; Jiachen Tao; Changchang Sun; Zhongpai Gao; Zhenghao Zhao; Anwesa Choudhuri; Gengyu Zhang; Meng Zheng; Feiran Wang; Terrence Chen; Yan Yan; Ziyan Wu

Paper

Consistent Instance Field for Dynamic Scene Understanding

Abstract

We introduce Consistent Instance Field, a continuous and probabilistic spatio-temporal representation for dynamic scene understanding. Unlike prior methods that rely on discrete tracking or view-dependent features, our approach disentangles visibility from persistent object identity by modeling each space-time point with an occupancy probability and a conditional instance distribution. To realize this, we introduce a novel instance-embedded representation based on deformable 3D Gaussians, which jointly encode radiance and semantic information and are learned directly from input RGB images and instance masks through differentiable rasterization. Furthermore, we introduce new mechanisms to calibrate per-Gaussian identities and resample Gaussians toward semantically active regions, ensuring consistent instance representations across space and time. Experiments on HyperNeRF and Neu3D datasets demonstrate that our method significantly outperforms state-of-the-art methods on novel-view panoptic segmentation and open-vocabulary 4D querying tasks.