3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting

TL;DR

This work introduces 3DGUT, a rasterization-friendly framework that replaces the EWA-based projection in 3D Gaussian Splatting with the Unscented Transform. By using sigma points, 3DGUT can project Gaussian particles under arbitrary camera models, including rolling shutter, without Jacobian derivations, and it aligns the rendering with ray tracing to support secondary rays. It maintains real-time rendering speeds while providing enhanced fidelity for distorted cameras and time-varying projections, demonstrated across standard benchmarks and autonomous-driving datasets. The approach enables hybrid splatting-tracing workflows, enabling reflections, refractions, and complex lighting within a unified 3D Gaussian representation. Overall, 3DGUT broadens applicability of Gaussian particle scenes to practical, distortion-prone imaging scenarios with minimal loss in efficiency.

Abstract

3D Gaussian Splatting (3DGS) enables efficient reconstruction and high-fidelity real-time rendering of complex scenes on consumer hardware. However, due to its rasterization-based formulation, 3DGS is constrained to ideal pinhole cameras and lacks support for secondary lighting effects. Recent methods address these limitations by tracing the particles instead, but, this comes at the cost of significantly slower rendering. In this work, we propose 3D Gaussian Unscented Transform (3DGUT), replacing the EWA splatting formulation with the Unscented Transform that approximates the particles through sigma points, which can be projected exactly under any nonlinear projection function. This modification enables trivial support of distorted cameras with time dependent effects such as rolling shutter, while retaining the efficiency of rasterization. Additionally, we align our rendering formulation with that of tracing-based methods, enabling secondary ray tracing required to represent phenomena such as reflections and refraction within the same 3D representation. The source code is available at: https://github.com/nv-tlabs/3dgrut.

Figures (16)

• Figure 1: We extend 3D Gaussian Splatting (3DGS) to support nonlinear camera projections and secondary rays for simulating phenomena such as reflections and refractions. By replacing EWA splatting rasterization with the Unscented Transform, our approach retains real-time efficiency while accommodating complex camera effects like rolling shutter. (Left) A comparison of our model trained on undistorted views vs. the original distorted fisheye views, showing that training on the full set of pixels improves visual quality. (Right) Two synthetic objects, a reflective sphere and a refractive statue, inserted into a scene reconstructed with our model.
• Figure 2: When projecting a Gaussian particle from 3D space onto the camera image plane, Monte Carlo sampling (left) provides the most accurate estimate but is costly to compute. EWA Splatting formulation used in kerbl3Dgaussians approximates the projection function via linearization, which requires a dedicated Jacobian $J$ for each camera model and leads to approximation errors with increasing distortion. Unscented Transform instead approximates the particle with Sigma points than can be projected exactly and from which the 2D conic can then be estimated.
• Figure 3: For a given ray, 3DGS kerbl3Dgaussians evaluates the response of the Gaussian particle in 2D after the projection onto the camera image plane. This requires backpropagation through the (approximated) projection function. Instead, we follow 3dgrt2024 and evaluate particles in 3D at the point of the maximum response along the ray.
• Figure 4: Qualitative comparison of our novel-view synthesis results against the baselines on the MipNERF360 dataset barron2022mipnerf360.
• Figure 5: Comparison of our renderings against Fisheye-GS liao2024fisheye, on scenes from the Scannet++ dataset yeshwanthliu2023scannetpp.