Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh

TL;DR

Mani-GS tackles editable photo-realistic rendering for 3D Gaussian Splatting by binding Gaussians to a triangle-local space and self-adapting to mesh manipulation. The triangle-shape aware binding enables large deformations, local edits, and soft-body simulations while preserving high rendering fidelity, even with imperfect meshes. The approach delivers state-of-the-art results on standard datasets and demonstrates robust performance across static rendering, manipulation, and simulation scenarios. This work significantly enhances the practicality of editable 3DGS for content creation and dynamic scenes.

Abstract

Neural 3D representations such as Neural Radiance Fields (NeRF), excel at producing photo-realistic rendering results but lack the flexibility for manipulation and editing which is crucial for content creation. Previous works have attempted to address this issue by deforming a NeRF in canonical space or manipulating the radiance field based on an explicit mesh. However, manipulating NeRF is not highly controllable and requires a long training and inference time. With the emergence of 3D Gaussian Splatting (3DGS), extremely high-fidelity novel view synthesis can be achieved using an explicit point-based 3D representation with much faster training and rendering speed. However, there is still a lack of effective means to manipulate 3DGS freely while maintaining rendering quality. In this work, we aim to tackle the challenge of achieving manipulable photo-realistic rendering. We propose to utilize a triangular mesh to manipulate 3DGS directly with self-adaptation. This approach reduces the need to design various algorithms for different types of Gaussian manipulation. By utilizing a triangle shape-aware Gaussian binding and adapting method, we can achieve 3DGS manipulation and preserve high-fidelity rendering after manipulation. Our approach is capable of handling large deformations, local manipulations, and soft body simulations while keeping high-quality rendering. Furthermore, we demonstrate that our method is also effective with inaccurate meshes extracted from 3DGS. Experiments conducted demonstrate the effectiveness of our method and its superiority over baseline approaches.

Figures (10)

• Figure 1: Our proposed approach allows for 3DGS manipulation, including large deformation, local manipulation, and even physical simulation (such as soft body), while maintaining high-quality rendering. (Check https://gaoxiangjun.github.io/mani_gs/ for more visual results.)
• Figure 2: Overview of our method.(1) Firstly, we extract a triangular mesh from 3DGS kerbl20233d or a neural surface field(NeuS wang2021neus). (2) Next, we bind $N$ Gaussians to each triangle in the local triangle space, and optimize the local Gaussian attributes ($\bm {\mu^l, R^l, s^l, o, c}$). The triangle attributes ($\bm{\mu^t, R^t, e}$) are calculated based on the triangle vertices. (3) Finally, we manipulate 3DGS by transferring the mesh manipulation directly, thus achieving manipulable rendering.
• Figure 3: Visual comparison between ours, NeRF-Editing liu2021editing(N.E.) and SuGaR guedon2023sugar for static rendering. It illustrates our proposed method can contain a much more accurate boundary in "Ship", and distinct results in "Drums".
• Figure 4: We offer an editing comparison between our method, SuGaR, and NeuMesh. Our approach demonstrates fewer artifacts and less blurring effects than SuGaR, and presents more abundant and distinct details compared to NeuMesh. For further details, please zoom in.
• Figure 5: Visual comparison of inaccurate mesh binding and rendering. Note that ours and SuGaR employ the same underlying mesh extracted from poisson recon.