MeshSplats: Mesh-Based Rendering with Gaussian Splatting Initialization

TL;DR

MeshSplats, a method which converts Gaussian Splatting to a mesh-like format, enabling rendering using ray tracing methods with all their associated benefits and enhancing the reconstruction quality through the application of a dedicated optimization algorithm that operates on mesh faces rather than Gaussian components.

Abstract

Gaussian Splatting (GS) is a recent and pivotal technique in 3D computer graphics. GS-based algorithms almost always bypass classical methods such as ray tracing, which offers numerous inherent advantages for rendering. For example, ray tracing is able to handle incoherent rays for advanced lighting effects, including shadows and reflections. To address this limitation, we introduce MeshSplats, a method which converts GS to a mesh-like format. Following the completion of training, MeshSplats transforms Gaussian elements into mesh faces, enabling rendering using ray tracing methods with all their associated benefits. Our model can be utilized immediately following transformation, yielding a mesh of slightly reduced quality without additional training. Furthermore, we can enhance the reconstruction quality through the application of a dedicated optimization algorithm that operates on mesh faces rather than Gaussian components. The efficacy of our method is substantiated by experimental results, underscoring its extensive applications in computer graphics and image processing.

Figures (8)

• Figure 1: MeshSplats (our) facilitates the conversion of Gaussian Splatting into a mesh format, which can subsequently be rendered using prevailing tools such as Blender and Nvdiffrast. This transformation enables sophisticated lighting effects, including reflections and shadows, as well as the capacity to execute mesh-based simulations.
• Figure 2: Transformation of a flat Gaussian into a mesh by MeshSplats (our), where $m$ is the mean of the Gaussian, $r_2$ and $r_3$ are the second and third columns of the rotation matrix $R$ (see Eq. \ref{['eq:cov']}), while $sr_1$ and $sr_2$ denote $\text{scaled\_rot}_1$ and $\text{scaled\_rot}_2$ from Eq. \ref{['eq:2d_scaledrot']}.
• Figure 3: Examples of renderings produced by MeshSplats (our), with lighting conditions and mesh deformations applied using Blender or Nvdiffrast. The rendered objects exhibit various lighting effects, with the top-middle render showcasing an object with a blended texture. The underlying original images (not shown here) were taken from the NeRF Synthetic dataset.
• Figure 4: Transformation of a 3D Gaussian into a mesh by MeshSplats (our). Here $m$ is the mean of the Gaussian, $r_1$, $r_2$, and $r_3$ are columns of the rotation matrix $R$, and $sr_1$, $sr_2$, and $sr_3$ represent $\text{scaled\_rot}_1$, $\text{scaled\_rot}_2$, and $\text{scaled\_rot}_3$ from Eq. \ref{['eq:3d_scaledrot']}. The transformation is performed analogously to the flat case (see Fig. \ref{['fig:modle']}), but it is extended to the three surfaces spanned by the vector pairs $(r_1,r_2)$, $(r_1,r_3)$, and $(r_2,r_3)$. Meshes are created for each surface and combined into a single mesh representing the 3D Gaussian.
• Figure 5: Examples of renderings of different types: the first column shows the ground truth image, the second column shows the rendering of the optimized MeshSplats (our) using Nvdiffrast, and the third column shows the rendering of 3D Gaussian Splatting with spherical harmonics set to zero. The first two rows comprise data from the Mip-NeRF360 dataset, where both MeshSplats and 3DGS were optimized with the resolution parameter set to 4. The last row is composed of the Truck example from the Tanks and Templates dataset, where both MeshSplat and 3DGS were optimized with the resolution parameter set to one.