Gated Fields: Learning Scene Reconstruction from Gated Videos

TL;DR

Gated Fields introduces a differentiable neural rendering approach that reconstructs outdoor scene geometry and radiance from active time-gated video sequences. By explicitly modeling gated image formation and jointly learning geometry, ambient illumination, material properties, and gating parameters as neural fields, the method leverages depth cues in gated captures to achieve dense, illumination-robust reconstructions. It demonstrates superior depth, 3D reconstruction, and novel-view synthesis performance versus RGB-only and LiDAR-guided baselines, across day and night conditions, on a new large outdoor dataset. The work advances practical large-scale scene understanding for autonomous systems, with code and data released for reproducibility and future enhancements.

Abstract

Reconstructing outdoor 3D scenes from temporal observations is a challenge that recent work on neural fields has offered a new avenue for. However, existing methods that recover scene properties, such as geometry, appearance, or radiance, solely from RGB captures often fail when handling poorly-lit or texture-deficient regions. Similarly, recovering scenes with scanning LiDAR sensors is also difficult due to their low angular sampling rate which makes recovering expansive real-world scenes difficult. Tackling these gaps, we introduce Gated Fields - a neural scene reconstruction method that utilizes active gated video sequences. To this end, we propose a neural rendering approach that seamlessly incorporates time-gated capture and illumination. Our method exploits the intrinsic depth cues in the gated videos, achieving precise and dense geometry reconstruction irrespective of ambient illumination conditions. We validate the method across day and night scenarios and find that Gated Fields compares favorably to RGB and LiDAR reconstruction methods. Our code and datasets are available at https://light.princeton.edu/gatedfields/.

Figures (5)

• Figure 1: From a single video of gated captures (top-row), we reconstruct an accurate scene representation and render depth projections (mid-row, right) as accurate as LiDAR scans (mid-row, left), and we recover 3D geometry and normals (bottom-row).
• Figure 2: Gated Image Formation and Bi-Directional Sampling. Top-row: Our test vehicle is equipped with a synchronized stereo camera setup and illuminator that flood-lits the scene with a light pulse and FoV $\gamma$. Using different gating profiles $C(z)$, we capture three slices with intensity visualised here in red, green and blue. Illustrated in the middle row, the gating profiles describe pixel intensity for a point at sensor distance $z$. The first slice (red) accounts for close ranges, the green for mid-ranges, and the blue for far ranges. Bottom-row: we show the ray sampling employed in our method, based on a bidirectional sampling strategy. We cast the rays from the illuminator view to explore the occluded areas, while the rays casted from the camera integrate the reflected scene response. The shadowed areas are marked in gray.
• Figure 3: Neural Gated Fields. For any point in space $\mathbf{x}$, we learn its volumetric density $\sigma$, normal $\mathbf{n}$, reflectance $\alpha$ and ambient lighting $\Lambda$ through four neural fields, conditioned on direction $\mathbf{d}$, incident laser light direction $\mathbf{\omega}$ and spatial embedding $\chi$. The illuminator light $\iota$ is represented by a physics-based model dependent on the displacement angle $\gamma$, while the gating imaging process is described by the range intensity profiles $\tilde{C}(z)$ using as input the camera-point-laser distance $z$, as explained in \ref{['sec:gated_imaging_model']}. With this information, we reconstruct a gated image $I_k$ through the gated volume rendering formulation introduced in \ref{['gated_field_learning']}. As this process is fully differentiable, we simultaneously fit neural fields and physical parameters through image reconstruction together with other regularization losses discussed in \ref{['training_supervision']}.
• Figure 4: Top: Example captures from our collected dataset across different urban and suburban areas in North America. From left to right: RGB image, active gated slices (with red for slice 1, green for slice 2 and blue for slice 3), passive slice, projected LiDAR scan, accumulated LiDAR. Bottom: Sensors setup with LiDAR, stereo Gated camera, stereo RGB camera, IMU and GNSS.
• Figure 5: Qualitative comparison of the proposed Gated Fields and state-of-the-art depth estimation approaches, including LiDAR-NeRF tao2023lidarnerf, StreetSurf guo2023streetsurf, NLSPN park2020nonRGBLidar, and Gated Stereo gatedstereo. Compared to baseline methods, we are able to reconstruct fine geometry details like branches or poles, also for far distances. Unlike RGB methods, Gated Fields is unaffected by poor ambient lighting, and unlike LiDAR-based methods it is able to reconstruct sharp object discontinuities. The active gated slices are visualized in red for slice 1, green for slice 2 and blue for slice 3.