REdiSplats: Ray Tracing for Editable Gaussian Splatting

TL;DR

Gaussian Splatting (GS) models 3D scenes as a set of Gaussians with trainable color and opacity, enabling fast rendering but lacking relighting, shadows, and physical interaction. The paper introduces REdiSplats, a differentiable framework that fuses ray tracing with a mesh approximation of flat 3D Gaussians to enable relighting, reflections, manual edits, and physical simulation, while remaining compatible with Blender and Nvdiffrast. Innovations include a planar octagon mesh representation for flat Gaussians, efficient ray–Gaussian intersection via NVIDIA OptiX, and a color-accumulation scheme along rays that combines learned opacity with the Gaussian density, i.e. $\alpha_i \approx \hat{\alpha}_i e^{-\frac{1}{2} \| o' + \hat{t} d' \|^2}$, enabling physically plausible lighting and editing. Across Mip-NeRF360, Tanks and Temples, and Deep Blending, REdiSplats yields competitive reconstruction quality and, notably, state-of-the-art performance on Deep Blending, while enabling rendering in Blender and Nvdiffrast and supporting manual edits and physics-based interactions, thus bridging GS and traditional graphics pipelines.

Abstract

Gaussian Splatting (GS) has become one of the most important neural rendering algorithms. GS represents 3D scenes using Gaussian components with trainable color and opacity. This representation achieves high-quality renderings with fast inference. Regrettably, it is challenging to integrate such a solution with varying light conditions, including shadows and light reflections, manual adjustments, and a physical engine. Recently, a few approaches have appeared that incorporate ray-tracing or mesh primitives into GS to address some of these caveats. However, no such solution can simultaneously solve all the existing limitations of the classical GS. Consequently, we introduce REdiSplats, which employs ray tracing and a mesh-based representation of flat 3D Gaussians. In practice, we model the scene using flat Gaussian distributions parameterized by the mesh. We can leverage fast ray tracing and control Gaussian modification by adjusting the mesh vertices. Moreover, REdiSplats allows modeling of light conditions, manual adjustments, and physical simulation. Furthermore, we can render our models using 3D tools such as Blender or Nvdiffrast, which opens the possibility of integrating them with all existing 3D graphics techniques dedicated to mesh representations.

Figures (10)

• Figure 1: REdiSplats model allows to model varying light conditions, including shadows, light reflections and mirror reflection. Moreover, we can edit the scene manually or using a physical ending.
• Figure 2: REdiSplats incorporates ray tracing into the 3D Gaussian Splatting framework. The figure presents a physical simulation model rendered in Blender.
• Figure 3: REdiSplats (our) uses ray-tracing-based approach. We need two points with regard to rays passing through Gaussian distributions. We use flat GS to utilize mesh to approximate such primitives, with the mesh-ray intersection being very efficient.
• Figure 4: REdiSplats allows to model mirror reflection and light conditioning thanks to ray tracing model.
• Figure 5: REdiSplats use ray tracing ending instead of restoration. Therefore, we can build scenes from GS and add a mesh-based element that interacts with the scene.