Quanta Video Restoration

TL;DR

This paper introduces Quanta Video Restoration (QUIVER), an end-to-end trainable network built on the core ideas of classical quanta restoration methods, i.e., pre-filtering, flow estimation, fusion, and refinement.

Abstract

The proliferation of single-photon image sensors has opened the door to a plethora of high-speed and low-light imaging applications. However, data collected by these sensors are often 1-bit or few-bit, and corrupted by noise and strong motion. Conventional video restoration methods are not designed to handle this situation, while specialized quanta burst algorithms have limited performance when the number of input frames is low. In this paper, we introduce Quanta Video Restoration (QUIVER), an end-to-end trainable network built on the core ideas of classical quanta restoration methods, i.e., pre-filtering, flow estimation, fusion, and refinement. We also collect and publish I2-2000FPS, a high-speed video dataset with the highest temporal resolution of 2000 frames-per-second, for training and testing. On simulated and real data, QUIVER outperforms existing quanta restoration methods by a significant margin. Code and dataset available at https://github.com/chennuriprateek/Quanta_Video_Restoration-QUIVER-

Figures (11)

• Figure 1: Goal of this paper. (a) Blur-free video frame of a moving car. (b)-(d) CMOS image sensor simulations using realistic sensor parameters. The strong shot noise and read noise ($5.1\;\text{e}^-$/pix) of CMOS sensor make the signal acquisition difficult. (e) With low read noise ($0.2\;\text{e}^-$/pix), low-bit single-photon detectors capture valuable information. (f)-(g) Existing state-of-the-art algorithm, QBP maQuantaBurstPhotography2020 cannot handle strong motion and noise. (h) The proposed algorithm, QUIVER, produces high quality results.
• Figure 2: Motion Blur and SNR Trade-off. The effects of bit depth on SNR and motion blur are illustrated using real captures by a single-photon sensor. For the motion range we target, $3$-bit single-photon detectors provide the best trade-off between blur and SNR. The images are captured using a $1$-bit SPAD duttonSPADBasedQVGAImage2016 at $10$k fps at an average photon level of $0.51$ and $0.40$ photons-per-pixel (PPP) per frame, respectively. Higher bit-depth outputs are generated through temporal frame averaging.
• Figure 3: Traditional Methods' Design. Depiction of existing classical quanta restoration algorithms' design philosophy. Best viewed in zoom.
• Figure 4: Traditional Methods' Limitations. (a) Traditional methods' maQuantaBurstPhotography2020gyongySinglePhotonTrackingHighSpeed2018 predenoising/temporal-averaging fails in strong motion. It is visible in the restored images that an input with strong motion between the frames results in several artifacts in the output even though SNR levels are similar. (b) Traditional methods maQuantaBurstPhotography2020 utilize a patch-based pre-trained optical flow module similar to hasinoffBurstPhotographyHigh2016b. This optical flow module fails to compensate for motion in the presence of noise.
• Figure 5: The proposed QUIVER network. The corresponding stages of QUIVER, built by embracing the intuitive thoughts behind existing classical methods. Best viewed in zoom.