A comprehensive study of time-of-flight non-line-of-sight imaging

Abstract

Time-of-Flight non-line-of-sight (ToF NLOS) imaging techniques provide state-of-the-art reconstructions of scenes hidden around corners by inverting the optical path of indirect photons scattered by visible surfaces and measured by picosecond resolution sensors. The emergence of a wide range of ToF NLOS imaging methods with heterogeneous formulae and hardware implementations obscures the assessment of both their theoretical and experimental aspects. We present a comprehensive study of a representative set of ToF NLOS imaging methods by discussing their similarities and differences under common formulation and hardware. We first outline the problem statement under a common general forward model for ToF NLOS measurements, and the typical assumptions that yield tractable inverse models. We discuss the relationship of the resulting simplified forward and inverse models to a family of Radon transforms, and how migrating these to the frequency domain relates to recent phasor-based virtual line-of-sight imaging models for NLOS imaging that obey the constraints of conventional lens-based imaging systems. We then evaluate performance of the selected methods on hidden scenes captured under the same hardware setup and similar photon counts. Our experiments show that existing methods share similar limitations on spatial resolution, visibility, and sensitivity to noise when operating under equal hardware constraints, with particular differences that stem from method-specific parameters. We expect our methodology to become a reference in future research on ToF NLOS imaging to obtain objective comparisons of existing and new methods.

Figures (13)

• Figure 1: (a) ToF NLOS imaging setups measure multiply scattered photons following paths between a laser device at $\mathbf{x}_0$, a laser target $\mathbf{x}_1$, sensor target $\mathbf{x}_{n-1}$, and sensor device at $\mathbf{x}_n$. Classic approaches assume perfect calibration of laser and sensor devices and targets (red and blue), third-bounce-only occlusion-free transport (green), and diffuse surface reflectance (purple). Under these assumptions, time-resolved NLOS measurements can be modeled by different types of Radon transforms. (b) NLOS light transport of photons with time of flight $t_{lv}+t_{sv}$ under unconstrained $\mathbf{x}_l$ and $\mathbf{x}\xspace_{s}\xspace$ is modeled by the elliptical Radon transform (ERT), whose inverse is typically approximated via filtered backprojection (\ref{['sec:ERT']}). (c) By assuming planar locality at surface points $\mathbf{x}_v$, NLOS light transport can be approximated by the planar Radon transform (PRT), whose inverse can be solved analytically by means of Laplacian-filtered backprojection (\ref{['sec:PRT']}). (d) Confocal setups co-locate $\mathbf{x}_l \equiv \mathbf{x}\xspace_{s}\xspace$, under which the ERT becomes a spherical Radon transform (SRT); existing methods leveraged this constraint to implement efficient solvers for the inverse SRT (\ref{['sec:PRT']}).
• Figure 2: (a) Top view of the simulated scene made of a $1m\times1m$ patch co-planar and at 1 meter from the relay wall, under confocal acquisition. (b) Reconstructions obtained at increasing photon counts (columns) for every method (rows). (c,d) PSNR and MS-SSIM metrics of all the methods for the simulated photon counts ($x$ axis). Point markers correspond to the reconstructions shown in (b).
• Figure 3: (a) Simulated scene with a $1m\times1m$ patch at 1 meter from the relay wall, confocal capture. (b) XZ frequency spectrum of the unfiltered backprojection output. (c) Front view (XY) of the reconstructions analyzed methods. (d) XZ frequency spectrum of the reconstructions yielded by each method. (e) Frequency spectrum of our estimated filtering operator for each method (\ref{['eq:estimated_filter']}).
• Figure 4: Variations on the width of the LoG filter, from $4$ to $19cm$. Wider filters skew the reconstructions towards lower frequencies, which mitigates noise but fails to correctly reproduce surface edges.
• Figure 5: Variations on the LCT parameter $\alpha$ which controls response to SNR on the Wiener filtering. Lower alpha values mitigate noise at the expense of blurring the edges of the reconstructed surface.