Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wallcamera: Reinventing the Wheel?

Aurélien Bourquard, Jeff Yan

TL;DR

The paper analyzes Wallcamera through the lens of Differential Imaging Forensics (DIF), arguing that Wallcamera reproduces the DIF core idea of extracting and amplifying latent signals from indirect light interactions but achieves finer activity granularity with CNN-based recognition. It contrasts the two approaches across concepts, methods, and experiments, highlighting similarities in differential signal extraction and differences in space-space processing and ML usage. The authors emphasize that DIF encompasses a broader forensic toolkit, including biometric leakage and deepfake detection, beyond mere activity recognition, and note that Wallcamera provides independent validation of DIF ideas albeit without full citation of prior work. The discussion also situates both approaches within non-line-of-sight imaging, outlines practical considerations such as camera resolution and processing spaces, and calls for proper attribution and cross-domain collaboration to advance NLOS forensics.

Abstract

Developed at MIT CSAIL, the Wallcamera has captivated the public's imagination. Here, we show that the key insight underlying the Wallcamera is the same one that underpins the concept and the prototype of differential imaging forensics (DIF), both of which were validated and reported several years prior to the Wallcamera's debut. Rather than being the first to extract and amplify invisible signals -- aka latent evidence in the forensics context -- from wall reflections in a video, or the first to propose activity recognition following that approach, the Wallcamera's actual innovation is achieving activity recognition at a finer granularity than DIF demonstrated. In addition to activity recognition, DIF as conceived has a number of other applications in forensics, including 1) the recovery of a photographer's personal identifiable information such as body width, height, and even the color of their clothing, from a single photo, and 2) the detection of image tampering and deepfake videos.

Wallcamera: Reinventing the Wheel?

TL;DR

The paper analyzes Wallcamera through the lens of Differential Imaging Forensics (DIF), arguing that Wallcamera reproduces the DIF core idea of extracting and amplifying latent signals from indirect light interactions but achieves finer activity granularity with CNN-based recognition. It contrasts the two approaches across concepts, methods, and experiments, highlighting similarities in differential signal extraction and differences in space-space processing and ML usage. The authors emphasize that DIF encompasses a broader forensic toolkit, including biometric leakage and deepfake detection, beyond mere activity recognition, and note that Wallcamera provides independent validation of DIF ideas albeit without full citation of prior work. The discussion also situates both approaches within non-line-of-sight imaging, outlines practical considerations such as camera resolution and processing spaces, and calls for proper attribution and cross-domain collaboration to advance NLOS forensics.

Abstract

Developed at MIT CSAIL, the Wallcamera has captivated the public's imagination. Here, we show that the key insight underlying the Wallcamera is the same one that underpins the concept and the prototype of differential imaging forensics (DIF), both of which were validated and reported several years prior to the Wallcamera's debut. Rather than being the first to extract and amplify invisible signals -- aka latent evidence in the forensics context -- from wall reflections in a video, or the first to propose activity recognition following that approach, the Wallcamera's actual innovation is achieving activity recognition at a finer granularity than DIF demonstrated. In addition to activity recognition, DIF as conceived has a number of other applications in forensics, including 1) the recovery of a photographer's personal identifiable information such as body width, height, and even the color of their clothing, from a single photo, and 2) the detection of image tampering and deepfake videos.
Paper Structure (11 sections, 3 figures)

This paper contains 11 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Concepts and Results
  3. DIF
  4. The Wallcamera
  5. Comparisons and Discussions
  6. Experiments: settings, configurations and cameras
  7. Non-line-of-sight (NLOS) aspect
  8. Signal extraction and amplification
  9. Parameter estimation
  10. Machine learning
  11. Conclusions

Figures (3)

  • Figure 1: The gist of DIF in a video-experiment setting similar to the Wallcamera's, reproduced here from Figure 3 in Bourquard and Yan dif (published in 2019). (a) Schematics of the scene setting with a camera pointed towards the wall. The space between the wall and the camera is sufficient for the passage of an intruder in-between. (b) Available scene frames (top row) and corresponding difference scene frames (bottom row). From left to right: (1) the intruder stands outside the apartment with the door closed, (2) the intruder has just entered the apartment, (3) the intruder approaches a plastic white board, and (4) the intruder stands in front of the plastic-white-board surface. All the video frames and contents thereof are best seen magnified in an e-copy.
  • Figure 2: The gist of the Wallcamera. Note: both this figure and the following caption are reproduced from Figure 3 in Sharma et al iccv21 (published in 2021). (a) A representative frame of the seemingly static input video. (b) A frame of the amplified residual video after subtracting the mean frame reveals faint changes in illumination caused by motion of the people. (c) A sequence of frames shows the motion of these features.
  • Figure 3: Schematic setup of the Wallcamera, reproduced from Sharma et al iccv21 (i.e., its Figure 2a).