SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting

TL;DR

SplatMesh tackles the challenge of fine-grained 3D editing under memory constraints by proposing a hybrid representation that fuses a downsampled mesh with 3D Gaussian splats. The pipeline builds a colored mesh from multi-view data, performs geometry- and color-aware downsampling, and tightly couples Gaussian splats to mesh geometry, enabling coherent view synthesis and interactive segmentation/ editing driven by 2D prompts. The approach introduces a 3D segmentation framework that extends 2D prompts to 3D and two editing modalities—geometry deformation and texture painting—both of which propagate consistently to the splats across views; ablations demonstrate the benefits of geometry-texture fusion and memory-guided sampling. Across real and synthetic datasets, SplatMesh achieves superior rendering quality and editing performance while using substantially fewer 3D points, highlighting its practical impact for VR/AR content creation and interactive 3D tooling.

Abstract

A key challenge in fine-grained 3D-based interactive editing is the absence of an efficient representation that balances diverse modifications with high-quality view synthesis under a given memory constraint. While 3D meshes provide robustness for various modifications, they often yield lower-quality view synthesis compared to 3D Gaussian Splatting, which, in turn, suffers from instability during extensive editing. A straightforward combination of these two representations results in suboptimal performance and fails to meet memory constraints. In this paper, we introduce SplatMesh, a novel fine-grained interactive 3D segmentation and editing algorithm that integrates 3D Gaussian Splat with a precomputed mesh and could adjust the memory request based on the requirement. Specifically, given a mesh, \method simplifies it while considering both color and shape, ensuring it meets memory constraints. Then, SplatMesh aligns Gaussian splats with the simplified mesh by treating each triangle as a new reference point. By segmenting and editing the simplified mesh, we can effectively edit the Gaussian splats as well, which will lead to extensive experiments on real and synthetic datasets, coupled with illustrative visual examples, highlighting the superiority of our approach in terms of representation quality and editing performance. Code of our paper can be found here: https://github.com/kaichen-z/SplatMesh.

Figures (7)

• Figure 1: Illustration. Our method enables high-quality interactive 3D segmentation and editing for multi-view representations. The top row displays the original images, while the row below presents the edited results, where the target book has been enlarged and stylized into a cartoon-like appearance.
• Figure 2: Illustration of Mesh Construction and Surface Rendering in SplatMesh. The SplatMesh involves 2 steps. In Step (1), a multi-view approach is employed to reconstruct the mesh, followed by projecting the color onto the mesh to create the colored mesh representation. In Step (2), the colored mesh is initially downsampled based on geometry and texture information, and 3DGS is then sampled within the new set of vertices.
• Figure 3: Illustration of 3D Segmentation Pipeline. The red and green stars represent $\bm{pr}^{oth}_{new}$ and $\bm{pr}^{obj}_{new}$, used for the SAM model.
• Figure 4: Qualitative Comparison of View Synthesis on NeRF Synthetic Dataset. We conducted a comparative analysis with SuGaR, evaluating the outcomes of extracting 3DGS from meshes and rendering test perspectives under identical fine-tuning iterations. Our* denotes the configuration employing our mesh downsampling method with a 1/3 face retention ratio and Our denotes the configuration w/o using downsampling. In contrast to SuGaR, our model demonstrates a significant reduction in artifact generation and yields results with enhanced granularity.
• Figure 5: Example of 3D Segmentation with SplatMesh. Our approach enables interactive 3D segmentation using 3D prompts. The first column displays the positions of 2D prompts, while the remaining columns present the 3D segmentation results.