2D Triangle Splatting for Direct Differentiable Mesh Training

Abstract

Differentiable rendering with 3D Gaussian primitives has emerged as a powerful method for reconstructing high-fidelity 3D scenes from multi-view images. While it offers improvements over NeRF-based methods, this representation still encounters challenges with rendering speed and advanced rendering effects, such as relighting and shadow rendering, compared to mesh-based models. In this paper, we propose 2D Triangle Splatting (2DTS), a novel method that replaces 3D Gaussian primitives with 2D triangle primitives. This representation naturally forms a discrete mesh-like structure while retaining the benefits of continuous volumetric modeling. Through the incorporation and controlled annealing of a compactness parameter, our method maintains differentiability during training while producing triangle meshes with fully opaque faces at the end of optimization without the need for additional post-processing. Experimental results demonstrate that our triangle-based representation achieves competitive visual quality with Gaussian-based methods while providing a more direct bridge to mesh-based representations. Our method bridges the gap between differentiable rendering and traditional mesh-based rendering, offering a promising solution for applications requiring renderable mesh-like reconstructions. Please visit our project page at https://gaoderender.github.io/triangle-splatting.

Figures (7)

• Figure 1: 2D Triangle Splatting for Mesh Reconstruction. Visualization of a model optimized using our proposed 2D Triangle Splatting method on the NeRF-Synthetic Ship scene.
• Figure 2: Overview of the proposed 2D Triangle Splatting (2DTS) method. For each pixel on the rendered image, we calculate the intersection between the pixel ray and each triangle primitive in the 3D space. The opacity and color of each triangle's contribution to the pixel is dependent on the barycentric coordinates of the intersection point. The final color of the pixel is calculated by blending the colors of all triangles that cover the pixel. A compactness parameter $\mathbf{\gamma}$ is introduced to control the sharpness of the triangle edges. Panel (a) shows a triangle with $\mathbf{\gamma} = 2$, and panel (b) shows a triangle with $\mathbf{\gamma} = 50$.
• Figure 3: Demonstration of Meshes Reconstructed by 2DTS. A visualization of meshes reconstructed by our method on the DTU jensen2014large dataset. From left to right: colored image rendered by our triangle splatting renderer, mesh rendered by Blender's Eevee engine, and normal map rendered by our renderer.
• Figure 4: Comparison of Mesh Extraction. Visual comparison of meshes reconstructed by our method, 2DGS Huang2DGS2024, GOF Yu2024GOF, PGSR Chen2024PGSR, and Nvdiffrec Munkberg_2022_CVPR on the NeRF-Synthetic mildenhall2020nerf dataset. Our method better preserves finer structures such as the leaves of the ficus, the metal grille of the mic, and the ropes of the ship.
• Figure 5: Auxiliary Chart for the Integration of Opacity.