Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RaySplats: Ray Tracing based Gaussian Splatting

Krzysztof Byrski, Marcin Mazur, Jacek Tabor, Tadeusz Dziarmaga, Marcin Kądziołka, Dawid Baran, Przemysław Spurek

TL;DR

RaySplats addresses the lighting limitations of 3D Gaussian Splatting by introducing a differentiable ray-tracing based framework that operates directly on Gaussian ellipsoids represented as $\mathcal{N}(\mathbf{m}_i,\Sigma_i)$. It computes ray-ellipse intersections via a Mahalanobis-distance formulation and aggregates color along rays with a forward/backward pass that stores intersected Gaussians for efficient gradients, enabling light, shadows, and reflections and seamless mesh integration. The method uses a RGB loss $\mathcal{L} = (1-\lambda) \mathcal{L}_2 + \lambda \mathcal{L}_{D-SSIM}$ and demonstrates competitive quantitative results while providing qualitatively improved lighting and material effects on standard datasets. Across Mip-NeRF360, Tanks and Temples, and Deep Blending, RaySplats achieves high-fidelity reconstructions and supports complex scenes with glass, shadows, and reflections, illustrating a practical path toward photorealistic, mesh-supported Gaussian rendering. Overall, RaySplats expands the applicability of Gaussian Splatting to realistic lighting scenarios with efficient, differentiable ray tracing.

Abstract

3D Gaussian Splatting (3DGS) is a process that enables the direct creation of 3D objects from 2D images. This representation offers numerous advantages, including rapid training and rendering. However, a significant limitation of 3DGS is the challenge of incorporating light and shadow reflections, primarily due to the utilization of rasterization rather than ray tracing for rendering. This paper introduces RaySplats, a model that employs ray-tracing based Gaussian Splatting. Rather than utilizing the projection of Gaussians, our method employs a ray-tracing mechanism, operating directly on Gaussian primitives represented by confidence ellipses with RGB colors. In practice, we compute the intersection between ellipses and rays to construct ray-tracing algorithms, facilitating the incorporation of meshes with Gaussian Splatting models and the addition of lights, shadows, and other related effects.

RaySplats: Ray Tracing based Gaussian Splatting

TL;DR

RaySplats addresses the lighting limitations of 3D Gaussian Splatting by introducing a differentiable ray-tracing based framework that operates directly on Gaussian ellipsoids represented as . It computes ray-ellipse intersections via a Mahalanobis-distance formulation and aggregates color along rays with a forward/backward pass that stores intersected Gaussians for efficient gradients, enabling light, shadows, and reflections and seamless mesh integration. The method uses a RGB loss and demonstrates competitive quantitative results while providing qualitatively improved lighting and material effects on standard datasets. Across Mip-NeRF360, Tanks and Temples, and Deep Blending, RaySplats achieves high-fidelity reconstructions and supports complex scenes with glass, shadows, and reflections, illustrating a practical path toward photorealistic, mesh-supported Gaussian rendering. Overall, RaySplats expands the applicability of Gaussian Splatting to realistic lighting scenarios with efficient, differentiable ray tracing.

Abstract

3D Gaussian Splatting (3DGS) is a process that enables the direct creation of 3D objects from 2D images. This representation offers numerous advantages, including rapid training and rendering. However, a significant limitation of 3DGS is the challenge of incorporating light and shadow reflections, primarily due to the utilization of rasterization rather than ray tracing for rendering. This paper introduces RaySplats, a model that employs ray-tracing based Gaussian Splatting. Rather than utilizing the projection of Gaussians, our method employs a ray-tracing mechanism, operating directly on Gaussian primitives represented by confidence ellipses with RGB colors. In practice, we compute the intersection between ellipses and rays to construct ray-tracing algorithms, facilitating the incorporation of meshes with Gaussian Splatting models and the addition of lights, shadows, and other related effects.

Paper Structure

This paper contains 25 sections, 2 theorems, 82 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Novel-View Synthesis
  4. Ray-Tracing Methods
  5. RaySplats -- Ray Tracing based Gaussian Splatting
  6. 3D Gaussian Splatting
  7. Intersection of the Gaussian Distribution with the Ray
  8. Color Aggregation Along the Ray
  9. Loss Function of the RaySplats Model
  10. Experiments
  11. Datasets and Metrics
  12. Quantitative Results
  13. Qualitative Results
  14. Ablation Study
  15. Implementation Details
  16. ...and 10 more sections

Key Result

Proposition 3.1

Consider the ray defined as ${\bf r}(t) = {\bf o} + t {\bf d}$, where ${\bf o}$ is the origin and ${\bf d}$ is the direction, and the Gaussian component $\mathcal{N}({\bf \mu}, \Sigma)$. The first (closest to the origin ${\bf o}$) intersection between the ray ${\bf r}(t)$ and the confidence ellipse where

Figures (8)

  • Figure 1: RaySplats (our) incorporates ray tracing into the 3D Gaussian Splatting framework. This allows us to integrate meshes with lighting conditions and mirror effects.
  • Figure 2: RaySplats (our) uses ray-tracing based solutions. In practice, we need two important points on rays passing through Gaussian distributions. Then, the maximum response point is utilized for aggregating colors along each ray. On the other hand, the intersection of Gaussian confidence ellipses is used to efficiently detect Gaussians with non-empty intersection with the ray.
  • Figure 3: RaySplats (our) allows us to combine 3D Gaussian splatting with a mesh-based rendering using lighting effects such as shadows and transparency.
  • Figure 4: RaySplats (our) allows us to combine 3D Gaussian splatting with a mesh-based rendering using lighting effects such as mirror reflections.
  • Figure 5: RaySplats (our) is capable of modeling glass elements in the 3D Gaussian Splatting environment, thereby facilitating the accurate visualization of glass reflections and the distortion of light due to refraction.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Proposition 3.1: Following hearn2010computer
  • Proposition 3.2: Following moenne20243d