MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering

TL;DR

This work proposes a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes and achieves a reduction in the number of Gaussian splats by 30% compared to the original 3D Gaussian splatting.

Abstract

Recently, 3D Gaussian splatting has gained attention for its capability to generate high-fidelity rendering results. At the same time, most applications such as games, animation, and AR/VR use mesh-based representations to represent and render 3D scenes. We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. In particular, we introduce a distance-based Gaussian splatting technique to align the Gaussian splats with the mesh surface and remove redundant Gaussian splats that do not contribute to the rendering. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound Gaussian splats. The tightly-bound splats are flattened and aligned well with the mesh geometry. The loosely-bound Gaussian splats are used to account for the artifacts in reconstructed 3D meshes in terms of rendering. We present a training strategy of binding Gaussian splats to the mesh geometry, and take into account both types of splats. In this context, we introduce several regularization techniques aimed at precisely aligning tightly-bound Gaussian splats with the mesh surface during the training process. We validate the effectiveness of our method on large and unbounded scene from mip-NeRF 360 and Deep Blending datasets. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR, particularly on the outdoor mip-NeRF 360 dataset, demonstrating better rendering quality. We provide analyses for each type of Gaussian splat and achieve a reduction in the number of Gaussian splats by 30% compared to the original 3D Gaussian splatting.

Figures (11)

• Figure 1: Our Approach: In Section \ref{['sec:GeometryReconstruction']}, we jointly train the geometry $f_{sdf}$ and appearance $g_{color}$ using differentiable volumetric rendering (shown in orange). Then, we extract the lightweight mesh $M$ which is important for the initialization of Gaussian splats. Next, in Section \ref{['sec:Mesh-based Gaussian Splatting']}., we present the mesh-based Gaussian splatting for training, which involves the removal of Gaussian splats (green Gaussian splats) occluded by mesh surface. Given the distance $d_{i}$ between each Gaussian splat and its corresponding triangle, we distinguish between tightly-bound Gaussian splats (blue splat) and loosely Gaussian splats (red splat). In Section \ref{['sec:Training']}, we introduce the training and densification strategy for both tightly-bound Gaussian splats and loosely Gaussian splats.
• Figure 2: The examples of extracted mesh from BakedSDF yariv2023bakedsdf, SuGaR guedon2023sugar, and LTM Choi2024LTM. We employ LTM Choi2024LTM to extract a lightweight mesh with moderate triangle count, using it for the initialization of Gaussian splats. The primary reason is that LTM utilizes a minimal number of triangles while maintaining geometric quality. Zoomed-in regions visualize that the mesh frequently lacks highly detailed geometry or complex structures. "K" and "M" denotes the $10^3$ and $10^6$ units, respectively.
• Figure 3: Qualitative Comparisons with Existing Methods. We visually compare our method with SuGaR guedon2023sugar, 3DGS kerbl20233d, and LTM Choi2024LTM. For a detailed description, we visualize the zoomed-in blue and red regions.
• Figure 4: Qualitative Comparisons of Our Method and SuGaR guedon2023sugar. We show shading mesh without texture.
• Figure 5: The number of Gaussian Splats The unit for the number of Gaussian splats is $10^3$. Compared to the original 3DGS, we utilize 30% fewer Gaussian splats. MeshGS* did not utilize any regularization techniques to tightly align Gaussian splats; instead, it solely applied image loss $L_{img}$ for training. MeshGS-(a) and MeshGS*-(c) shows the number of tightly-bound Gaussian splats. MeshGS-(b) and MeshGS*-(d) denotes the number of loosely-bound Gaussian splats.