Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting

TL;DR

This work tackles data efficiency in semantic and dynamic 3D scene understanding by formulating next-best-view (NBV) selection as active learning powered by Fisher Information. It introduces a unified 3D Gaussian Splatting backbone that jointly handles semantic reasoning and temporal deformation, with a diagonal FI approximation for scalability and a trace-based estimator for FI of the deformation network. The proposed method outperforms random and uncertainty-based baselines on static Replica and dynamic Neu3D datasets in both rendering quality and semantic segmentation. By enabling information-driven view acquisition, the approach offers a principled, scalable path toward open-vocabulary, dynamic semantic 3D scene understanding for robotics, AR/VR, and content creation.

Abstract

Understanding semantics and dynamics has been crucial for embodied agents in various tasks. Both tasks have much more data redundancy than the static scene understanding task. We formulate the view selection problem as an active learning problem, where the goal is to prioritize frames that provide the greatest information gain for model training. To this end, we propose an active learning algorithm with Fisher Information that quantifies the informativeness of candidate views with respect to both semantic Gaussian parameters and deformation networks. This formulation allows our method to jointly handle semantic reasoning and dynamic scene modeling, providing a principled alternative to heuristic or random strategies. We evaluate our method on large-scale static images and dynamic video datasets by selecting informative frames from multi-camera setups. Experimental results demonstrate that our approach consistently improves rendering quality and semantic segmentation performance, outperforming baseline methods based on random selection and uncertainty-based heuristics.

Figures (4)

• Figure 1: Overview of our work. At each timestep, we select the NBV from a set of candidate camera views using a Fisher Information-based criterion. The selection process evaluates each candidate view based on its expected contribution to both semantic feature learning and dynamic deformation modeling. Once selected, the chosen view is incorporated into the training process of our Dynamic and Semantic 3DGS. For rendering and supervision, the Gaussians are first deformed by an MLP deformation network based on the current timestep, then projected into RGB images and semantic feature maps. These outputs are compared against ground-truth images and semantic labels to jointly optimize both the dynamic deformation and semantic representation.
• Figure 2: Semantic segmentation results on Replica dataset. The first and third rows of images are the rendered results. The second and forth rows of images are the results of visualizing the feature maps of test set after being trained with 12 training views. All the methods are based on Feature 3DGS feature3dgs except for different view selection methods to augment the training data.
• Figure 3: Zoomed-in qualitative study of our method on Neu3D dataset. The second and fourth rows are zoom-in figures. Visualizations are the results of the test set after being trained with 39 training views. All the methods are based on the same dynamic and semantic 3DGS, except for different view selection methods to augment the training data.
• Figure 4: Heatmap for selection method interest on Neu3D dataset. The second row is zoom-in figures.