Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft Shadow Diffusion (SSD): Physics-inspired Learning for 3D Computational Periscopy

Fadlullah Raji, John Murray-Bruce

TL;DR

This work tackles the challenge of reconstructing both a hidden occluder in 3D and a hidden non-occluding plane from a single penumbra photograph in passive NLOS imaging. It introduces a physics-inspired, separable nonlinear least squares formulation and offers two inversion paths: a gradient-based optimizer and a diffusion-prior based physics-inspired neural network called Soft Shadow Diffusion (SSD). SSD learns a high-fidelity 3D occluder point cloud from soft shadow data, which is then used to obtain the 2D non-occluder image via TV-regularized fitting and mesh extraction via an SDF-based mesh generator. Across synthetic and real experiments, the methods demonstrate accurate reconstructions and robustness to noise and ambient illumination, generalizing beyond the shapes seen during training and highlighting the practical potential of passive NLOS 3D imaging.

Abstract

Conventional imaging requires a line of sight to create accurate visual representations of a scene. In certain circumstances, however, obtaining a suitable line of sight may be impractical, dangerous, or even impossible. Non-line-of-sight (NLOS) imaging addresses this challenge by reconstructing the scene from indirect measurements. Recently, passive NLOS methods that use an ordinary photograph of the subtle shadow cast onto a visible wall by the hidden scene have gained interest. These methods are currently limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. Here, we generalize this class of methods and demonstrate a 3D reconstruction of a hidden scene from an ordinary NLOS photograph. To achieve this, we propose a novel reformulation of the light transport model that conveniently decomposes the hidden scene into \textit{light-occluding} and \textit{non-light-occluding} components to yield a separable non-linear least squares (SNLLS) inverse problem. We develop two solutions: A gradient-based optimization method and a physics-inspired neural network approach, which we call Soft Shadow diffusion (SSD). Despite the challenging ill-conditioned inverse problem encountered here, our approaches are effective on numerous 3D scenes in real experimental scenarios. Moreover, SSD is trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. SSD also shows surprising robustness to noise and ambient illumination.

Soft Shadow Diffusion (SSD): Physics-inspired Learning for 3D Computational Periscopy

TL;DR

This work tackles the challenge of reconstructing both a hidden occluder in 3D and a hidden non-occluding plane from a single penumbra photograph in passive NLOS imaging. It introduces a physics-inspired, separable nonlinear least squares formulation and offers two inversion paths: a gradient-based optimizer and a diffusion-prior based physics-inspired neural network called Soft Shadow Diffusion (SSD). SSD learns a high-fidelity 3D occluder point cloud from soft shadow data, which is then used to obtain the 2D non-occluder image via TV-regularized fitting and mesh extraction via an SDF-based mesh generator. Across synthetic and real experiments, the methods demonstrate accurate reconstructions and robustness to noise and ambient illumination, generalizing beyond the shapes seen during training and highlighting the practical potential of passive NLOS 3D imaging.

Abstract

Conventional imaging requires a line of sight to create accurate visual representations of a scene. In certain circumstances, however, obtaining a suitable line of sight may be impractical, dangerous, or even impossible. Non-line-of-sight (NLOS) imaging addresses this challenge by reconstructing the scene from indirect measurements. Recently, passive NLOS methods that use an ordinary photograph of the subtle shadow cast onto a visible wall by the hidden scene have gained interest. These methods are currently limited to 1D or low-resolution 2D color imaging or to localizing a hidden object whose shape is approximately known. Here, we generalize this class of methods and demonstrate a 3D reconstruction of a hidden scene from an ordinary NLOS photograph. To achieve this, we propose a novel reformulation of the light transport model that conveniently decomposes the hidden scene into \textit{light-occluding} and \textit{non-light-occluding} components to yield a separable non-linear least squares (SNLLS) inverse problem. We develop two solutions: A gradient-based optimization method and a physics-inspired neural network approach, which we call Soft Shadow diffusion (SSD). Despite the challenging ill-conditioned inverse problem encountered here, our approaches are effective on numerous 3D scenes in real experimental scenarios. Moreover, SSD is trained in simulation but generalizes well to unseen classes in simulation and real-world NLOS scenes. SSD also shows surprising robustness to noise and ambient illumination.
Paper Structure (40 sections, 19 equations, 21 figures, 2 tables, 4 algorithms)

This paper contains 40 sections, 19 equations, 21 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Computational Periscopy:
  4. Shape from S:
  5. Diffusion Probabilistic Models:
  6. Light Propagation Model
  7. Hidden Scene Decomposition and Computational FOV
  8. 3D Computational Periscopy: Models & Representation
  9. A Simple Pinspeck Occluder
  10. Uniform Voxels Representation
  11. Point Cloud Representation
  12. Soft Shadow Diffusion Model
  13. 3D Computational Periscopy: Inversion
  14. Inversion via Gradient-based Optimization
  15. Implementation details:
  16. ...and 25 more sections

Figures (21)

  • Figure 1: NLOS imaging configuration. Non-occluding objects diffusely reflect light toward the visible wall; the reflected light is partially occluded by an occluding object in the hidden scene to create a soft shadow on the visible wall.
  • Figure 1: Each panel shows the variation of the visibility function over the visible wall/camera FOV for some patch in the hidden-scene light-emitting 2D plane and a hidden-scene pinspeck voxel. The visibility function is binary-valued, with one (yellow) indicating that the patch in the hidden-scene light-emitting 2D plane is unoccluded from those patches on the visible wall, and zero (dark blue) indicating occlusion by the pinspeck voxel.
  • Figure 2: Computational FOV and two simple examples of occluders. (a) The camera's FOV is projected towards the non-occluding scene plane. The occluding region is the volume created by joining the boundaries of the measurement FOV to the boundaries of the desired hidden scene plane. (b) An ideal pinhole projects an image of the scene. (c) An ideal pinspeck projects a negative image of the scene.
  • Figure 2: Increased resolution reconstructions in simulations using sparse data structures.
  • Figure 3: Soft Shadow Diffusion Model Pipeline. (a) In training, the model maps a clean input point cloud to noise using the forward diffusion process. (b) At inference, the model starts from an initial noise and transforms this noise into a point cloud representation of light-occluding structure by conditioning on the encoded soft shadow photograph $\hat{\mathbf{y}}$. The point cloud is then converted into a mesh representation of the occluding structure.
  • ...and 16 more figures