Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

Nicolas Moenne-Loccoz, Ashkan Mirzaei, Or Perel, Riccardo de Lutio, Janick Martinez Esturo, Gavriel State, Sanja Fidler, Nicholas Sharp, Zan Gojcic

TL;DR

A specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order is described, which comes at little cost compared to rasterization.

Abstract

Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.

3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

TL;DR

A specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order is described, which comes at little cost compared to rasterization.

Abstract

Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.
Paper Structure (58 sections, 8 equations, 18 figures, 6 tables)

This paper contains 58 sections, 8 equations, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Novel-View Synthesis and Neural Radiance Fields
  4. Point-Based and Particle Rasterization
  5. Differentiable Ray Tracing of Volumetric Particles
  6. Background
  7. 3D Gaussian Parameterization
  8. Differentiable Rendering of Particle Representations
  9. Hardware-Accelerated Ray Tracing
  10. Method
  11. Bounding Primitives
  12. Stretched Polyhedron Proxy Geometry
  13. Ray Tracing Renderer
  14. Motivation
  15. Algorithm
  16. ...and 43 more sections

Figures (18)

  • Figure 1: Runtime Ray Tracing Effects: Our ray-based pipeline is easily compatible with conventional ray-based visual effects at test time, including reflections (top left), depth of field (top middle), refractions (bottom left), hard shadows cast by meshes (bottom middle), and myriad combinations of them (right).
  • Figure 2: Overview of the Accelerated Tracing Algorithm: Given a set of 3D particles, we first build the corresponding bounding primitives and insert them into a BVH. To compute the incoming radiance along each ray, we trace rays against the BVH to get the next k particles. We then compute the intersected particles' response and accumulate the radiance according to Equation \ref{['eq:QuadratureVolumeRendering']}. The process repeats until all particles have been evaluated or the transmittance meets a predefined threshold and the final rendering is returned.
  • Figure 3: Proxy Geometries: Examples of BVH primitives considered.
  • Figure 4: Rendering: on each round of tracing, the next $k$ closest hit particles are collected and sorted into depth order along the ray, the radiance is computed in-order, and the ray is cast again to collect the next hits.
  • Figure 5: Particle Kernel Functions: (a) In addition to 3D Gaussians, in this work we investigated three other particle types: the generalized Gaussian (GG2), kernelized surface (SGG2) and cosine wave modulation (CSGG2) particles. (b) Shows radiance and normal reconstructions obtained with the kernelized surface particles for two scenes.
  • ...and 13 more figures