Neural Surface Priors for Editable Gaussian Splatting

TL;DR

The paper tackles editable scene reconstruction by marrying a neural surface prior with 3D Gaussian Splatting. It first learns a neural SDF using PermutoSDF to produce a high-quality mesh, which then guides the placement and appearance of Gaussian components, with opacity tied to distance to the surface via $\sigma(oldsymbol{x}) = (oldsymbol{eta} ext{ function} ext{ composed with } f_ heta)(oldsymbol{x})$ and $oldsymbol{eta}$ learned during training. To enable intuitive edits, Gaussians are encoded into a triangle soup proxy, allowing mesh edits to propagate to the recovered appearance through a linear transform between the original and edited mesh bases, yielding updated Gaussians while preserving visual fidelity. The approach demonstrates strong novel-view synthesis performance and supports topology-aware editing, including mesh-based modifications and physics-informed deformations, while maintaining rendering quality across mesh resolutions. Limitations include the need for a structurally sound, fixed-topology mesh and potential lighting-related inconsistencies; future work may address topology changes and relighting to further enhance realism.

Abstract

In computer graphics and vision, recovering easily modifiable scene appearance from image data is crucial for applications such as content creation. We introduce a novel method that integrates 3D Gaussian Splatting with an implicit surface representation, enabling intuitive editing of recovered scenes through mesh manipulation. Starting with a set of input images and camera poses, our approach reconstructs the scene surface using a neural signed distance field. This neural surface acts as a geometric prior guiding the training of Gaussian Splatting components, ensuring their alignment with the scene geometry. To facilitate editing, we encode the visual and geometric information into a lightweight triangle soup proxy. Edits applied to the mesh extracted from the neural surface propagate seamlessly through this intermediate structure to update the recovered appearance. Unlike previous methods relying on the triangle soup proxy representation, our approach supports a wider range of modifications and fully leverages the mesh topology, enabling a more flexible and intuitive editing process. The complete source code for this project can be accessed at: https://github.com/WJakubowska/NeuralSurfacePriors.

Figures (6)

• Figure 2: Schematic overview of our pipeline. (1) Starting with a collection of input images and corresponding camera poses, the initial stage utilizes PermutoSDF to generate a neural SDF and extract a mesh. (2) The second stage involves training a Gaussian Splatting model. The initial locations of kernels are sampled directly from mesh, and the opacity is defined by the minimum distance from the surface. (3) Given the modified mesh, the third stage involves the extraction of a proxy triangle soup representation, which allows for the propagation of the geometric changes onto this proxy. By recovering the updated Gaussian parameters from this modified proxy, a revised appearance representation is obtained.
• Figure 3: Visualization of propagating the mesh edit to a single proxy triangle. We start by connecting the triangle $\mathbf{V}$ to the nearest mesh face (pictured in light blue). Then, its coordinates are expressed in the associated basis $\mathbf{U}$ (represented in the picture by gray axes pointing in the directions given by the basis). A subsequent transformation aligns the triangle in the modified basis $\mathbf{U}'$, resulting in an updated representation of a single Gaussian’s shape and position.
• Figure 4: A modification created with our method on a low-resolution mesh. The left side shows the unmodified mesh and its render, while the right displays the modified version with the excavator's bucket lowered. Despite the lower mesh resolution and applied edits, the output maintains high visual quality, comparable to the original high-resolution mesh.
• Figure 5: Comparison of rendering outputs for different Gaussian initialization schemes. Random initialization introduces artifacts, whereas sampling initial locations from the extracted mesh surface ensures cleaner and more accurate results.
• Figure 6: Progression of a wind-driven simulation of a Ficus plant. Top row: changes in the object's geometry. Bottom row: corresponding modifications in appearance obtained using our method.