BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream

TL;DR

BeNeRF addresses recovering a neural radiance field from a single blurry image paired with an event stream by jointly estimating the 3D scene and a continuous camera trajectory. It represents the scene with a NeRF expressed by an MLP and models motion via a differentiable cubic B-Spline in $SE(3)$, enabling synthesis of both the blur and brightness changes during exposure. A differentiable image-formation model combines blur and event data losses, allowing end-to-end optimization without ground-truth poses. Empirical results on synthetic and real data show BeNeRF yields view-consistent latent sharp images and competitive performance with multi-view event-based methods, while requiring only a single blurred image and a short event sequence.

Abstract

Neural implicit representation of visual scenes has attracted a lot of attention in recent research of computer vision and graphics. Most prior methods focus on how to reconstruct 3D scene representation from a set of images. In this work, we demonstrate the possibility to recover the neural radiance fields (NeRF) from a single blurry image and its corresponding event stream. We model the camera motion with a cubic B-Spline in SE(3) space. Both the blurry image and the brightness change within a time interval, can then be synthesized from the 3D scene representation given the 6-DoF poses interpolated from the cubic B-Spline. Our method can jointly learn both the implicit neural scene representation and recover the camera motion by minimizing the differences between the synthesized data and the real measurements without pre-computed camera poses from COLMAP. We evaluate the proposed method with both synthetic and real datasets. The experimental results demonstrate that we are able to render view-consistent latent sharp images from the learned NeRF and bring a blurry image alive in high quality. Code and data are available at https://github.com/wu-cvgl/BeNeRF.

Figures (10)

• Figure 1: Given a single blurry image and its corresponding event stream, BeNeRF can synthesize high-quality novel images along the camera trajectory, recovering a sharp and coherent video from the single blurry image.
• Figure 2: The pipeline of our method. Given a motion blurry image and its corresponding event stream, we aim to recover both the implicit sharp scene representation and its camera motion trajectory within exposure time. We exploit a continuous time representation for the motion trajectory, and maximize the coherence between both the real measurements and synthesized data for the recovery.
• Figure 2: Qualitative results of different methods with synthetic datasets. Detailed qualitative comparison for “Livingroom”, "Outdoorpool" and "Pinkcastle" scene of synthetic dataset.
• Figure 3: Qualitative results of different methods with synthetic datasets. It demonstrates that our method delivers better performance compared to prior approaches. The learning based methods fail to generalize on severely blurry images.
• Figure 3: Qualitative results of different methods with synthetic datasets. Detailed qualitative comparison for “Tanabata” and "Whiteroom" scene of synthetic dataset.