GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting

TL;DR

GaMeS addresses the conditioning and editability challenge of Gaussian Splatting by introducing mesh-based and mesh-free representations that anchor Gaussians to mesh faces or Triangle Soup. By parameterizing Gaussian means and covariances through mesh vertices and face geometry, GaMeS enables real-time modification and animation while preserving rendering quality. The method supports both mesh-initialized inputs and mesh-free Triangle Soup, producing competitive PSNR/SSIM/LPIPS results on NeRF-Synthetic and Mip-NeRF360 and enabling intuitive object edits and animations. Overall, GaMeS provides a flexible, efficient framework for editable 3D rendering that integrates mesh-aware Gaussian components with dynamic transformations.

Abstract

Gaussian Splatting (GS) is a novel, state-of-the-art technique for rendering points in a 3D scene by approximating their contribution to image pixels through Gaussian distributions, warranting fast training and real-time rendering. The main drawback of GS is the absence of a well-defined approach for its conditioning due to the necessity of conditioning several hundred thousand Gaussian components. To solve this, we introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes. We parameterize each Gaussian component by the vertices of the mesh face. Furthermore, our model needs mesh initialization on input or estimated mesh during training. We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation. As a result, we obtain a real-time rendering of editable GS.

Figures (19)

• Figure 1: GaMeS produce a hybrid of Gaussian Splatting (GS) and mesh representations. Therefore, GaMeS allows real-time modification and adaptation of GS.
• Figure 2: In the absence of a pre-existing input mesh, we parameterize Gaussians to create a structure referred to as Triangle Soup, which facilitates easy modification of the object. The Triangle Soup can be edited directly or modified using parametrization derived from estimated meshes, which do not need to be perfectly fitted. As a result, mesh-based deformation is applied to edit the object.
• Figure 3: GaMeS can be effectively trained on large scenes to allow their modifications while preserving high-quality renders.
• Figure 4: Visualization of affine transformation of Gaussian components when we modify mesh. Mesh vertices parameterize the mean and covariance matrix of Gaussian. Therefore, such parameters update when we change the mesh.
• Figure 5: The left image presents the Gaussian component constructed on the face. The right image presents how our model uses previously estimated Gaussian to construct $k$ (in Fig. $k=3$) components on the face.