Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment

TL;DR

This work tackles the problem of reconstructing sharp Neural Radiance Fields (NeRF) from motion-blurred images by jointly estimating the camera trajectory during exposure and the NeRF parameters. It introduces EBAD-NeRF, a framework that fuses blurred RGB frames with asynchronous event data through an intensity-change-metric event loss and a photometric blur loss, enabling end-to-end optimization in SE(3) for multiple poses per view. The method demonstrates that incorporating event information yields more accurate pose trajectories and sharper 3D reconstructions than prior image-based and ERGB-based deblurring NeRF approaches, both on synthetic and real-world data. The results indicate significant improvements in pose accuracy (ATE) and rendering quality, suggesting practical value for blur-prone imaging in dynamic 3D scene reconstruction applications.

Abstract

Neural Radiance Fields (NeRF) achieves impressive 3D representation learning and novel view synthesis results with high-quality multi-view images as input. However, motion blur in images often occurs in low-light and high-speed motion scenes, which significantly degrades the reconstruction quality of NeRF. Previous deblurring NeRF methods struggle to estimate pose and lighting changes during the exposure time, making them unable to accurately model the motion blur. The bio-inspired event camera measuring intensity changes with high temporal resolution makes up this information deficiency. In this paper, we propose Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters by leveraging the hybrid event-RGB data. An intensity-change-metric event loss and a photo-metric blur loss are introduced to strengthen the explicit modeling of camera motion blur. Experiments on both synthetic and real-captured data demonstrate that EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.

Figures (10)

• Figure 1: Motivation of our proposed method. In a low-light scene or with a high-speed moving camera, motion blur usually occurs in the captured images. To reconstruct sharp NeRF from blurry images, Deblur-NeRF Deblur-NeRF uses a deformable sparse kernel to model the blur process. BAD-NeRF bad-nerf linearly interpolates the camera poses and jointly learns the start and end poses of camera motion. However, these methods are unable to model complex motion blur and lack supervision information during exposure time with only a photo-metric loss. With the proposed event-driven bundle adjustment, our EBAD-NeRF can leverage the event to recover the accurate camera trajectory and learn a sharper NeRF, resulting in sharper novel view image rendering than the previous methods.
• Figure 2: Overview of our method. For each view of the static scene, an image with motion blur is caused by the camera moving during the exposure time. We temporal evenly sample $p$ poses $\{\mathbf{P}_{i}\}_{i=1}^{p}$ on the motion trajectory and transfer them into SE(3) as learnable variables $\{\mathbf{T}_i\}_{i=1}^{p}$. With the poses, we model the blurry image and event generation in our network and jointly optimize the NeRF parameters $\theta$ and motion trajectory with the ERGB data to learn a sharp NeRF eventually.
• Figure 3: Qualitative ablation study on the Tanabata scene. EBAD-NeRF-linear and EBAD-NeRF-noe are defined in Sec. \ref{['sec:4.3']}. With event-driven bundle adjustment, the rendering results of learned NeRF are sharper and clearer.
• Figure 4: Qualitative ablation study of trajectory fitting accuracy on Tanabata scene. EBAD-NeRF-linear and EBAD-NeRF-noe are defined in Sec. \ref{['sec:4.3']}. EBAD-NeRF significantly fits the camera motion closer to the ground truth in the roll, pitch, and yaw metrics than all other methods.
• Figure 5: Evaluation on the influence of the number of the sampled poses $p$ and the weight parameter $\lambda$. The results are averages of reconstructed images on both deblurring and novel views of the Cozyroom scene. The red, green, and blue lines represent PSNR, SSIM, and LPIPS, respectively.