Gaussian Wave Splatting for Computer-Generated Holography

TL;DR

This work derives a closed-form solution for a 2D Gaussian-to-hologram transform that supports occlusions and alpha blending and derives an efficient approximation of the aforementioned process in the Fourier domain that is easily parallelizable and implemented using custom CUDA kernels.

Abstract

State-of-the-art neural rendering methods optimize Gaussian scene representations from a few photographs for novel-view synthesis. Building on these representations, we develop an efficient algorithm, dubbed Gaussian Wave Splatting, to turn these Gaussians into holograms. Unlike existing computer-generated holography (CGH) algorithms, Gaussian Wave Splatting supports accurate occlusions and view-dependent effects for photorealistic scenes by leveraging recent advances in neural rendering. Specifically, we derive a closed-form solution for a 2D Gaussian-to-hologram transform that supports occlusions and alpha blending. Inspired by classic computer graphics techniques, we also derive an efficient approximation of the aforementioned process in the Fourier domain that is easily parallelizable and implement it using custom CUDA kernels. By integrating emerging neural rendering pipelines with holographic display technology, our Gaussian-based CGH framework paves the way for next-generation holographic displays.

Figures (6)

• Figure 1: Illustration of coordinate systems and Gaussian Wave Splatting.Top: We take a set of optimized primitives from off-the-shelf neural rendering frameworks in world space and convert them into into hologram space through a sequence of coordinate transformations. Each Gaussian in hologram space is represented by its own affine transformation from canonical space. Bottom: In hologram space, GWS computes the analytical solution of the wavefront footprint at the SLM generated by a Gaussian wavefront---with arbitrary rotation and position---illuminated by a plane wave.
• Figure 2: Order-invariant transparency with view-dependent opacity. We ablate novel-view synthesis of our 2DGS-OIT model using view-agnostic opacity (clamping opacity to $(0, 1)$ using a sigmoid function as in naive Gaussian splatting) and view-dependent opacity represented with spherical harmonics. The naïve OIT model suffers from severe color leakage or even complete disappearance of objects. Conversely, the OIT model with view-dependent opacity greatly mitigates such artifacts, achieving image quality comparable to 2DGS.
• Figure 3: Simulated comparisons on wavefront blending strategies. We ablate wavefront blending strategies to reconstruct a scene represented by a collection of Gaussians in simulation. Naïvely adding up all the Gaussians without considering occlusion leads to oversaturated colors (left). State-of-the-art occlusion handling techniques matsushima2014silhouette employ a binary aperture to attenuate wavefront contributions from the rear based on the opacity of the front object, and do not appropriately handle smooth Gaussian falloffs and alpha blending (center left). Our alpha wave blending method (center right) performs exact alpha blending and faithfully reconstructs the 3D scene. On the right, we show the target image rendered with ray-based alpha blending.
• Figure 4: Simulated comparisons on parallax. RGB-D data lacks information about objects behind occluders, resulting in blank regions when viewed from other viewpoints. The silhouette-based method matsushima2014silhouette, designed for binary apertures, masks the wavefronts from the rear, leading to inaccurate wavefront masking and blending of continuous Gaussian profiles. Our alpha wave blending method effectively handles the occlusion of Gaussians, naturally blends them according to their smooth falloffs from all viewpoints, and best matches the target shown on the right.
• Figure 5: Experimentally captured focal stacks of holograms generated using different primitive-based CGH algorithms. For each CGH baseline, we show the target rendering of its input 3D representation using a pinhole camera model (top row). Sparse point-cloud representations used in point-based methods chen2009computer inevitably lead to holes in the rendered target image. Polygon-based methods matsushima2009extremely produce overly smooth results unless an excessive number of polygons are used. Our Gaussian Wave Splatting methods achieve the best results with sharp in-focus regions and more accurate blur in defocus regions. Please refer to the supplementary materials for extensive baseline comparisons.