Mesh Splatting for End-to-end Multiview Surface Reconstruction

TL;DR

This work tackles the trade-off between volumetric representations (rich 3D context but challenging mesh extraction) and pure surface approaches (efficient but limited 3D context). It introduces mesh softening to turn a base mesh into a differentiable, multi-layer pseudo-volume that is rendered with a differentiable mesh splatting pipeline; this provides volumetric supervision while preserving mesh topology through a hybrid control scheme (DMTet early, Continuous Remeshing later). The approach achieves accurate surface reconstructions with substantially fewer vertices and shorter training times (about 20 minutes per scene) on object-centric datasets, and demonstrates competitive or superior performance with clear improvements in fine geometry and thin structures. Limitations include scalability to very large scenes and extremely thin features, motivating adaptive layer-widths and more flexible remeshing in future work. Overall, this method offers a practical bridge between volumetric and surface representations for efficient, high-quality surface reconstruction guided by image data.

Abstract

Surfaces are typically represented as meshes, which can be extracted from volumetric fields via meshing or optimized directly as surface parameterizations. Volumetric representations occupy 3D space and have a large effective receptive field along rays, enabling stable and efficient optimization via volumetric rendering; however, subsequent meshing often produces overly dense meshes and introduces accumulated errors. In contrast, pure surface methods avoid meshing but capture only boundary geometry with a single-layer receptive field, making it difficult to learn intricate geometric details and increasing reliance on priors (e.g., shading or normals). We bridge this gap by differentiably turning a surface representation into a volumetric one, enabling end-to-end surface reconstruction via volumetric rendering to model complex geometries. Specifically, we soften a mesh into multiple semi-transparent layers that remain differentiable with respect to the base mesh, endowing it with a controllable 3D receptive field. Combined with a splatting-based renderer and a topology-control strategy, our method can be optimized in about 20 minutes to achieve accurate surface reconstruction while substantially improving mesh quality.

Figures (9)

• Figure 1: Comparison of reconstruction paradigms. Yellow points denote ground-truth point clouds. "Verts" and "CD" denote the number of vertices and the Chamfer distance, respectively. (a) Our method optimizes meshes end-to-end and uses remeshing for topology control, achieving accurate surfaces with the fewest vertices. (b) SuGaR guedon2023sugar also optimizes meshes but relies on a single-layer Gaussian-splatting proxy and cannot perform remeshing, which limits accuracy. (c–d) As volumetric methods, GaussianSurfel dai2024high and Neuralangelo li2023neuralangelo require a meshing step to extract surfaces, which accumulates errors and often yields unnecessarily dense meshes; note the misalignment between their meshes and the point clouds (red circle). (e) Poisson reconstruction on the ground-truth points shows that even with accurate point clouds, meshing can still introduce errors—e.g., omission of points (blue circle)—which constrains the practical upper bound of volumetric pipelines.
• Figure 2: Comparison between regular meshes and soft mesh.
• Figure 3: Overview of the proposed method. An initial tetrahedral grid stores signed-distance values at its vertices, and a base mesh is extracted using Marching Tetrahedra. The base mesh is then softened into multiple layers by offsetting vertices along their normals, transforming it from a surface into a pseudo-volumetric representation. The multi-layer mesh is rendered via the proposed Differentiable Mesh Splatting based on tile-based rasterization, and supervised by the input images through a rendering loss.
• Figure 4: Qualitative comparison on DTU dataset.
• Figure 5: Qualitative comparison on BMVS dataset.