D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup

TL;DR

This paper tackles editing dynamic 3D scenes represented by Gaussian Splatting (GS) and proposes Dynamic Multi-Gaussian Soup (D-MiSo), a mesh-inspired, time-varying GS framework. It introduces Multi-Gaussians, comprising Core-Gaussians and Sub-Gaussians, parameterized by Triangle Soup, and employs two deformation networks to separately model global object motion and local rendering nuances. Training proceeds in two stages: Stage 1 initializes Core-Gaussians to capture motion; Stage 2 attaches and optimizes Sub-Gaussians (with a Sub-Rot Network) to achieve high-quality rendering and editable dynamics, with time $t$ governing updates. Experiments across D-NeRF, NeRF-DSyan2023nerfds, and PanopticSports show competitive reconstruction metrics and demonstrate intuitive editing capabilities, including object duplication, scaling, and dynamic remapping, highlighting D-MiSo’s practical impact for dynamic scene manipulation.

Abstract

Over the past years, we have observed an abundance of approaches for modeling dynamic 3D scenes using Gaussian Splatting (GS). Such solutions use GS to represent the scene's structure and the neural network to model dynamics. Such approaches allow fast rendering and extracting each element of such a dynamic scene. However, modifying such objects over time is challenging. SC-GS (Sparse Controlled Gaussian Splatting) enhanced with Deformed Control Points partially solves this issue. However, this approach necessitates selecting elements that need to be kept fixed, as well as centroids that should be adjusted throughout editing. Moreover, this task poses additional difficulties regarding the re-productivity of such editing. To address this, we propose Dynamic Multi-Gaussian Soup (D-MiSo), which allows us to model the mesh-inspired representation of dynamic GS. Additionally, we propose a strategy of linking parameterized Gaussian splats, forming a Triangle Soup with the estimated mesh. Consequently, we can separately construct new trajectories for the 3D objects composing the scene. Thus, we can make the scene's dynamic editable over time or while maintaining partial dynamics.

Figures (16)

• Figure 1: D-MiSo model parameterized dynamic scenes by Triangle Soup (disjoint triangles cloud), which allows modification of objects during time.
• Figure 2: Each object using the D-MiSo model is represented by Core-Gaussians and Sub-Gaussians, which form Multi-Gaussians. Each Gaussian is related to a triangle using parameterization proposed in GaMeS waczynska2024games. Triangles define the Gaussian shape (i.e., location, scale, rotation), and triangles clouds form Triangles Soups.
• Figure 3: D-MiSo allows us to modify scenes in similar ways as classical mesh-based models.
• Figure 4: Comparison of possible modifications in D-MiSo and the SC-GS. In the latter, authors use nodes while D-MiSo apply Sub-Triangle Soup (see the second column). We also must add static (pink) and dynamic (yellow) points in SC-GS to obtain modification by editing dynamic points. In practice, we have to use many static points to stop artifacts. Moreover, SC-GS is not an affine invariant and produces space when we change the size of the objects. In the case of D-MiSo, we marked points and applied modifications. Our model is superior in handling object scaling.
• Figure 5: One way to modify the object at the selected time $t_i$ is to take Core-Gaussians and apply a meshing strategy to obtain the correct mesh instead of Triangle Soup. Then, we can parametrize Sub-Gaussian in the coordinate system given bay mesh faces instead of Core-Triangle Soup. Finally, we can modify our mesh to obtain new modifications.