Tetrahedron Splatting for 3D Generation

TL;DR

This work introduces TeT-Splatting, a tetrahedron-based 3D representation that blends surface-based volumetric rendering with a structured tetrahedral grid to enable easy convergence, precise mesh extraction via Marching Tetrahedra, and real-time rendering using a tile-based differentiable rasterizer. The method incorporates eikonal and normal-consistency regularization for the signed distance field and supports training without mesh extraction, allowing seamless integration into existing diffusion-prior-driven 3D pipelines. Through a two-stage workflow, geometry is optimized with TeT-Splatting before texturing a polygonal mesh, achieving superior convergence speed, rendering efficiency, and mesh quality compared to NeRF-, DMTet-, and 3DGS-based approaches under both vanilla and rich diffusion priors. Rich priors further enhance fidelity and speed, yielding favorable CLIP-based metrics and faster geometry optimization, while preserving stable early-stage geometry. Overall, TeT-Splatting offers a practical, scalable solution for high-fidelity 3D content generation that can be integrated into existing 3D pipelines for applications in VR/AR, games, and design workflows.

Abstract

3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but is limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short in mesh extraction. In this work, we introduce a novel 3D representation, Tetrahedron Splatting (TeT-Splatting), that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within a structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT-Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT-Splatting strikes a superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.

Figures (13)

• Figure 1: 3D assets generated by our proposed TeT-Splatting.
• Figure 2: Left: An overview of TeT-Splatting. To produce the final renderings, we first pre-filter and remove nearly transparent tetrahedra, then project the remaining ones into 2D splats. These are blended based on opacity values derived from the SDF values at specific pixel intersections. Right: TeT-Splatting for 3D generation. We employ TeT-Splatting in the initial stage of the 3D generation pipeline and subsequently transition it to polygonal mesh for texture optimization.
• Figure 3: Normal map comparison during optimization of 3D generation. We utilize DMTet and TeT-Splatting as 3D representations in the geometry modeling stage of the RichDreamer qiu2023richdreamer. The first two rows show normal maps obtained from DMTet and TeT-Splatting during optimization. TeT-Splatting achieves more stable and smooth optimization, while DMTet becomes fragmented initially and gets stuck in an undesirable shape. The third row shows the normal maps of meshes extracted from the signed distance field of TeT-Splatting via Marching Tetrahedra shen2021dmtet (MT). As optimization progresses, TeT-Splatting's behavior aligns with rendering through MT.
• Figure 4: Qualitative comparison on 3D generation using vanilla RGB-based diffusion priors. We present visual comparisons of the rendered RGB maps and color maps from various 3D generation methods. The methods, arranged from left to right, are: Magic3D, Fantasia3D, DreamGaussian, and Ours. The comparison is conducted across two tasks, text-to-3D and image-to-3D, with results shown from top to bottom, respectively. Additionally, for each method, we provide the training time and the rendering speed (FPS) for the first stage of the process.
• Figure 5: Visualization of normal maps before and after mesh exportation. Note that the normal maps of DreamGaussian tang2023dreamgaussian are derived from its depth maps.