Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes

TL;DR

Dynamic Gaussians Mesh (DG-Mesh) addresses the challenge of obtaining high-fidelity, time-consistent meshes from dynamic observations. It builds on 3D Gaussian Splatting by learning deformable Gaussians in a canonical space, projecting them to per-frame surfaces via differentiable surface reconstruction, and enforcing cross-frame correspondences through Gaussian-Mesh Anchoring and cycle-consistent deformation. The method achieves superior mesh quality and rendering performance on synthetic and real dynamic datasets, and enables applications such as texture editing across time. It also introduces a practical training pipeline with end-to-end differentiable components suitable for integration into graphics pipelines.

Abstract

Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines. Project page: https://www.liuisabella.com/DG-Mesh

Figures (26)

• Figure 1: We propose DG-Mesh, a framework that reconstructs high-fidelity time-consistent mesh for dynamic scenes with complex non-rigid deformations. Given dynamic input and the camera parameters, our method reconstructs the high-quality surface and its appearance, as well as the mesh vertice motion across time frames. Our method can reconstruct mesh with flexible topology change as shown above. Additional results can be found on: https://www.liuisabella.com/DG-Mesh.
• Figure 2: The 3D Gaussian centers before and after the Gaussian-Mesh Anchoring.
• Figure 3: Main pipeline of DG-Mesh. We maintain a set of canonical 3D Gaussians. Under each time step, we transform it into a deformed space. We treat each set of deformed Gaussian points as an oriented point cloud and apply a differentiable Poisson Solver and differentiable Marching Cubes to recover the deformed surface. We propose Gaussian-Mesh Anchoring to adjust the deformed Gaussians to be uniformly aligned with the mesh faces. During anchoring, Gaussian densification and pruning are performed. We use a backward deformation module to project the newly adjusted Gaussian points back to the canonical space.
• Figure 4: Illustration of our Gaussian-Mesh Anchoring procedure. After obtaining mesh from the DPSR and differentiable Marching Cubes, we adjust the deformed 3D Gaussians to make it more aligned with the mesh's faces. For each Gaussian point, we find its nearest neighbors. If a mesh face is the nearest neighbor to multiple Gaussian points at the same time, we merge these Gaussians and create a new Gaussian from them. If a mesh face is not any Gaussian point's nearest neighbor, we create a new Gaussian on its center.
• Figure 5: We compare the mesh reconstruction and rendering results of our method with other baselines on the D-NeRF dataset. For each object, we visualize the results under two different time steps and different viewing angles. Our method outperforms others by producing more smooth and detailed structures.