Flying with Photons: Rendering Novel Views of Propagating Light

TL;DR

This work tackles rendering propagating light from novel moving viewpoints by introducing a transient-field neural rendering framework that couples a time-resolved radiance field $\boldsymbol{\tau}_\theta:(\mathbf{r},\mathbf{d})\mapsto \mathbb{R}_+^N$ with a volume density field $\sigma(\mathbf{r})$, capturing light-speed delays and view-dependent appearance. It leverages a first-of-its-kind multi-viewpoint transient dataset captured with a SPAD-based system, and trains an optimized volumetric representation to synthesize transient videos across viewpoints and time, including a time-warping capability to remove propagation delays and a path to relativistic effects. The method integrates SPAD photon-count models, a transient rendering equation with propagation delays, and an optimization procedure building on Instant-NGP to render both direct and global light transport with reflections, refractions, and diffraction. Applications demonstrated include time warping, relativistic rendering, and direct–global separation, highlighting potential impacts in education, visualization of ultrafast phenomena, and scientific imaging, while also acknowledging limitations due to static-scene capture times and the need for richer dynamic captures in future work.

Abstract

We present an imaging and neural rendering technique that seeks to synthesize videos of light propagating through a scene from novel, moving camera viewpoints. Our approach relies on a new ultrafast imaging setup to capture a first-of-its kind, multi-viewpoint video dataset with picosecond-level temporal resolution. Combined with this dataset, we introduce an efficient neural volume rendering framework based on the transient field. This field is defined as a mapping from a 3D point and 2D direction to a high-dimensional, discrete-time signal that represents time-varying radiance at ultrafast timescales. Rendering with transient fields naturally accounts for effects due to the finite speed of light, including viewpoint-dependent appearance changes caused by light propagation delays to the camera. We render a range of complex effects, including scattering, specular reflection, refraction, and diffraction. Additionally, we demonstrate removing viewpoint-dependent propagation delays using a time warping procedure, rendering of relativistic effects, and video synthesis of direct and global components of light transport.

Figures (7)

• Figure 1: Flying with Photons. The input to our method is a set of multi-viewpoint transient videos that capture a scene illuminated by a diffused or collimated pulsed light source. We then render videos of propagating light---transient videos---from different novel viewpoints at different moments in time. Left: Conventional image of the Kennedy and Coke bottle scenes. Centre: Grayscale transient frames rendered from novel viewpoints, composited over the colour image of the scene. Right: Adapting the peak-time visualization of Velten et al. velten2013femto, we show the entire transient video in a single frame. Hue indicates the time at which the peak intensity is observed at each pixel, and brightness corresponds to the magnitude of the peak intensity. We modulate the brightness over the time dimension to create color bands corresponding to equi-time paths (a.k.a isochrones), which reveal the shape of propagating wavefronts.
• Figure 2: We cast a ray into the scene and query a neural representation at samples $\mathbf{r}(s)$ along the ray to retrieve an $N$-dimensional transient $\boldsymbol{\tau}_\theta$ and density value $\sigma$ for each sample. Each element $\boldsymbol{\tau}_\theta[n]$ of that transient corresponds to a time bin of width $W$ times the speed of light $c$. We time-shift each transient based on the time delay from the camera origin $\mathbf{o}$ to $\mathbf{r}(s)$ and composite them together using volume rendering. The neural representation parameters are optimized to minimize the difference between the rendered transient $\boldsymbol{\tau}_\mathbf{r}$ and the captured transient $\boldsymbol{\tilde{\tau}}_\mathbf{r}$.
• Figure 3: Multi-viewpoint capture setup. A gantry controls the elevation angle of a scanning SPAD, and a scene and laser source are rotated together on a stage.
• Figure 4: Simulated results rendered from novel viewpoints for the peppers and smoke scenes. Rows 1, 4: The ground-truth integrated transient is shown alongside transient plots for all methods. The proposed method more accurately represents both the direct and global components of light. Rows 2, 5: For all methods, we show one frame of the transient video, composited over the integrated image of the scene. Rows 3, 6: Peak-time visualization illustrating the transient in a single frame. Hue encodes the time of peak intensity, brightness is modulated by the maximum intensity, and each band corresponds to an isochrone, or wavefront of equal path length.
• Figure 5: Captured results rendered from novel viewpoints for the David and mirror scenes. Row 1, 4: For all methods, we show one frame of the transient video, composited over the integrated image of the scene. Row 2, 5: Peak-time visualization. Row 3, 6: Transient plots. We show the captured photon count using a scatter plot instead of a continuous line, due to the sparse and quantized nature of the measurements. The methods reconstruct a continuous approximation of the underlying transient and suppress the noise observed in the captured data.