Iterating the Transient Light Transport Matrix for Non-Line-of-Sight Imaging

TL;DR

This work advances non-line-of-sight imaging by measuring the full Transient Light Transport Matrix (TLTM) on a relay surface with a 16×16 gated SPAD array and deriving a second-order TLTM, TLTM-2, for surfaces in the hidden scene. The authors develop fast, linear-algebra–based algorithms within the phasor-field framework to synthesize focused virtual illumination and extract TLTM-2 from TLTM-1, enabling applications such as scene relighting with novel or synthesized illumination, separation of direct and indirect light transport, and dual photography. They demonstrate substantial practical benefits, including parallel photon acquisition reducing data capture times and the ability to probe higher-order light transport phenomena like caustics and subsurface scattering. The approach offers a path to robust, high-resolution NLOS imaging and potentially higher-order TLTM reconstructions (TLTM-3, TLTM-4) as SPAD array pixel counts grow, with implications for disaster response, surveillance, and autonomous navigation.

Abstract

Active imaging systems sample the Transient Light Transport Matrix (TLTM) for a scene by sequentially illuminating various positions in this scene using a controllable light source, and then measuring the resulting spatiotemporal light transport with time of flight (ToF) sensors. Time-resolved Non-line-of-sight (NLOS) imaging employs an active imaging system that measures part of the TLTM of an intermediary relay surface, and uses the indirect reflections of light encoded within this TLTM to "see around corners". Such imaging systems have applications in diverse areas such as disaster response, remote surveillance, and autonomous navigation. While existing NLOS imaging systems usually measure a subset of the full TLTM, development of customized gated Single Photon Avalanche Diode (SPAD) arrays \cite{riccardo_fast-gated_2022} has made it feasible to probe the full measurement space. In this work, we demonstrate that the full TLTM on the relay surface can be processed with efficient algorithms to computationally focus and detect our illumination in different parts of the hidden scene, turning the relay surface into a second-order active imaging system. These algorithms allow us to iterate on the measured, first-order TLTM, and extract a \textbf{second order TLTM for surfaces in the hidden scene}. We showcase three applications of TLTMs in NLOS imaging: (1) Scene Relighting with novel illumination, (2) Separation of direct and indirect components of light transport in the hidden scene, and (3) Dual Photography. Additionally, we empirically demonstrate that SPAD arrays enable parallel acquisition of photons, effectively mitigating long acquisition times.

Figures (9)

• Figure 2: (A) Active Imaging System for measuring Transient Light Transport Matrix, which contains information about (i) Diffuse reflections (ii), specular reflections, and (iii) indirect components of Light Transport. (B) Non-line-of-sight Imaging System using Time of flight detectors is an active imaging system that measures the TLTM of the relay surface. (C) Existing acquisition schemes - Confocal and Non-Confocal - capture a subset of the Full Measurement. Multipixel arrays (Non-Confocal 2) can replicate data capture achieved by sequential laser scanning (Non-Confocal 1) while also reducing capture times through parallel acquisition of photons.
• Figure 3: Our NLOS imaging system consisting of laser, galvanometer, and SPAD Array. The right figure shows the SPAD pixels focused on the relay surface for large field of view, mimicking a virtual phased array system at the relay wall.
• Figure 4: (A) Hidden scene shown on the left. The non-transient, 3D reconstruction generated is shown in the middle (front view) and on the right (3D view). (B) We display 3 columns of TLTM-2 for three locations in the hidden scene from Panel(A). Row 1, 2, and 3 show the impact of focusing the wavefront at different 3D locations corresponding to letters $n$, W, and the mirror in the hidden scene. Columns 1 and 2 mark the focal spot using a blue and red cross respectively, and are generated by adding up all transient frames. Column 2 has the reconstruction overlaid on the ground truth image. Columns 3, 4, 5, 6 correspond to key frames within the video. (C) The TLTM is used to separate direct components (Columns 3,4) and indirect component (Columns 5, 6) of light when focusing the wavefront at locations corresponding to $n$ (Row 1) or W (Row 2). Columns 1 and 2 are generated by adding the direct and indirect components.
• Figure 5: (A) The left image shows the hidden scene while the middle image rough location for the illumination on the relay surface. The image on the right shows the front view of the 3D light field. Each single image is generated by averaging reconstructions over a 4x4 subgroup of neighboring SPAD pixels that share a TDC, and thus each image is labeled using the corresponding TDC index.(B) We demonstrate the impact of focusing a linear array on a point on the back wall, marked by a blue cross. Adding a spherical curvature to our illumination (Column 1) focuses to a vertical line. Interpolation (Column 2) to a finer illumination grid improves the resolution by enabling the use of a shorter phasor field wavelength. Solving an additional linear inverse and optimizing the illumination spatially reduces beamforming artifacts without improving the focus (Column 3). Column 4 shows that we can improve the spatial resolution with which we focus the virtual illumination significantly (Row 1) by sacrificing temporal coherence, which generates random illumination at other frames, as seen in Row 2. (C) By optimizing illumination over both time and space, we can generate incoherent point sources in the hidden scene with high spatial resolution. This allows us to project arbitrary patterns, like the "N" shown in the left image, onto the hidden scene, as displayed in the two images on the right.
• Figure S6: Qualitative comparison of SNR. Left, Top row: Single-pixel reconstructions at progressively longer exposure times. Left, Bottom row: Reconstructions with a fixed 7.25s exposure time using an increasing number of SPAD pixels. Right: Hidden object