Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions

TL;DR

Dblur e-NeRF is proposed, a novel method to directly and effectively reconstruct blur-minimal NeRFs from motion-blurred events generated under high-speed motion or low-light conditions and introduces a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.

Abstract

The stark contrast in the design philosophy of an event camera makes it particularly ideal for operating under high-speed, high dynamic range and low-light conditions, where standard cameras underperform. Nonetheless, event cameras still suffer from some amount of motion blur, especially under these challenging conditions, in contrary to what most think. This is attributed to the limited bandwidth of the event sensor pixel, which is mostly proportional to the light intensity. Thus, to ensure that event cameras can truly excel in such conditions where it has an edge over standard cameras, it is crucial to account for event motion blur in downstream applications, especially reconstruction. However, none of the recent works on reconstructing Neural Radiance Fields (NeRFs) from events, nor event simulators, have considered the full effects of event motion blur. To this end, we propose, Deblur e-NeRF, a novel method to directly and effectively reconstruct blur-minimal NeRFs from motion-blurred events generated under high-speed motion or low-light conditions. The core component of this work is a physically-accurate pixel bandwidth model proposed to account for event motion blur under arbitrary speed and lighting conditions. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches. Experiments on real and novel realistically simulated sequences verify our effectiveness. Our code, event simulator and synthetic event dataset will be open-sourced.

Figures (12)

• Figure 1: Existing works on NeRF reconstruction from moving event cameras heavily rely on (a). In contrast, Deblur e-NeRF is able to directly and effectively reconstruct blur-minimal NeRFs from (b), as shown in (c).
• Figure 2: Event motion blur from a white bar moving on a black background (From hu2021_v2e)
• Figure 3: Robust e-NeRF event generation model low2023_robust-e-nerf
• Figure 4: Core analog circuit of a typical event sensor pixel (Adapted from nozaki2017_temperature)
• Figure 5: Overview of the proposed pixel bandwidth model