Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NIGHT -- Non-Line-of-Sight Imaging from Indirect Time of Flight Data

Matteo Caligiuri, Adriano Simonetto, Pietro Zanuttigh

TL;DR

This paper tackles non-line-of-sight (NLoS) imaging using only indirect Time of Flight data from off-the-shelf itof sensors. The authors introduce the Mirror Trick, reframing the NLoS scene as a LOS problem by treating the front wall as a virtual mirror, which simplifies ground-truth generation and learning. A deep learning framework with an Encoder-Decoder architecture predicts mirrored itof measurements at $20\text{MHz}$ across a three-frequency input, enabling depth recovery of the hidden object via standard ToF relations. They build a fully synthetic 1344-scene dataset rendered with Mitsuba2, demonstrate feasible depth reconstruction with an average depth MAE of $5.21\text{ cm}$ and a mean intersection-over-union (miou) of $0.77$ on the test set, and show that multi-frequency inputs improve performance. The approach runs on full-field illumination for fast acquisition, offering a practical path toward real-time NLoS imaging with consumer ToF hardware, with future work aimed at validating on real data and enhancing realism of sensor noise models.

Abstract

The acquisition of objects outside the Line-of-Sight of cameras is a very intriguing but also extremely challenging research topic. Recent works showed the feasibility of this idea exploiting transient imaging data produced by custom direct Time of Flight sensors. In this paper, for the first time, we tackle this problem using only data from an off-the-shelf indirect Time of Flight sensor without any further hardware requirement. We introduced a Deep Learning model able to re-frame the surfaces where light bounces happen as a virtual mirror. This modeling makes the task easier to handle and also facilitates the construction of annotated training data. From the obtained data it is possible to retrieve the depth information of the hidden scene. We also provide a first-in-its-kind synthetic dataset for the task and demonstrate the feasibility of the proposed idea over it.

NIGHT -- Non-Line-of-Sight Imaging from Indirect Time of Flight Data

TL;DR

This paper tackles non-line-of-sight (NLoS) imaging using only indirect Time of Flight data from off-the-shelf itof sensors. The authors introduce the Mirror Trick, reframing the NLoS scene as a LOS problem by treating the front wall as a virtual mirror, which simplifies ground-truth generation and learning. A deep learning framework with an Encoder-Decoder architecture predicts mirrored itof measurements at across a three-frequency input, enabling depth recovery of the hidden object via standard ToF relations. They build a fully synthetic 1344-scene dataset rendered with Mitsuba2, demonstrate feasible depth reconstruction with an average depth MAE of and a mean intersection-over-union (miou) of on the test set, and show that multi-frequency inputs improve performance. The approach runs on full-field illumination for fast acquisition, offering a practical path toward real-time NLoS imaging with consumer ToF hardware, with future work aimed at validating on real data and enhancing realism of sensor noise models.

Abstract

The acquisition of objects outside the Line-of-Sight of cameras is a very intriguing but also extremely challenging research topic. Recent works showed the feasibility of this idea exploiting transient imaging data produced by custom direct Time of Flight sensors. In this paper, for the first time, we tackle this problem using only data from an off-the-shelf indirect Time of Flight sensor without any further hardware requirement. We introduced a Deep Learning model able to re-frame the surfaces where light bounces happen as a virtual mirror. This modeling makes the task easier to handle and also facilitates the construction of annotated training data. From the obtained data it is possible to retrieve the depth information of the hidden scene. We also provide a first-in-its-kind synthetic dataset for the task and demonstrate the feasibility of the proposed idea over it.
Paper Structure (27 sections, 5 equations, 6 figures, 2 tables)

This paper contains 27 sections, 5 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Analytical methods
  4. Deep Learning based methods
  5. Basic principles on Time of Flight imaging
  6. Indirect Time of Flight
  7. Direct Time of Flight Cameras
  8. NLoS perception using ToF cameras
  9. Proposed method
  10. The mirror trick
  11. Deep learning framework
  12. Input data
  13. Data augmentation
  14. Output data
  15. nn architecture
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Transient vector representation
  • Figure 2: Representation of the Mirror trick.
  • Figure 3: Schematized representation of the proposed nn model
  • Figure 4: Sample objects with their locations, orientations and dimensions.
  • Figure 5: Sample visual results
  • ...and 1 more figures