LSE-NeRF: Learning Sensor Modeling Errors for Deblured Neural Radiance Fields with RGB-Event Stereo

TL;DR

This work considers the camera modeling imperfections that arise from the simple pinhole camera model as learned embeddings for each camera measurement, and learns a mapper that connects event camera measurements with RGB data.

Abstract

We present a method for reconstructing a clear Neural Radiance Field (NeRF) even with fast camera motions. To address blur artifacts, we leverage both (blurry) RGB images and event camera data captured in a binocular configuration. Importantly, when reconstructing our clear NeRF, we consider the camera modeling imperfections that arise from the simple pinhole camera model as learned embeddings for each camera measurement, and further learn a mapper that connects event camera measurements with RGB data. As no previous dataset exists for our binocular setting, we introduce an event camera dataset with captures from a 3D-printed stereo configuration between RGB and event cameras. Empirically, we evaluate our introduced dataset and EVIMOv2 and show that our method leads to improved reconstructions. Our code and dataset are available at https://github.com/ubc-vision/LSENeRF.

Figures (6)

• Figure 1: Teaser -- We propose a deblur NeRF method that uses both RGB and event data. We focus on sensor modeling imperfections, which allows our method to effectively make use of both modalities. As shown, our method provides significantly sharper reconstructions compared to both when using only RGB and also other RGB/event NeRF baselines.
• Figure 2: Framework overview -- To render a pixel considering the camera motion blur, we pass points along the light rays of $n$ cameras through the hash grid to obtain their density and colors. We then volume render the pixel colors for the $n$ cameras and average them to generate the motion-blurred pixel color. We further utilize a learned mapper that maps RGB volume render to an intensity response for the event stream, which then utilizes the $n-1$ subsequent camera pairs to measure events. Note that our color Multi-Layer Perceptron (MLP) takes per-time learnable embeddings as input, to account for the sensor modeling imperfections.
• Figure 3: Sample RGB frames from each scene in our dataset -- Our dataset consists of (top row) five outdoor scenes and (bottom row) five indoor scenes. The substantial image blur is caused by rapid camera movements.
• Figure 4: Capture rig -- We 3D print a stereo casing that holds a GigE Blackfly S camera and a Prophesee EVK-3 HD camera.
• Figure 5: Qualitative examples -- We show qualitative examples of zoomed-in reconstruction cutouts. Our method provides the sharpest reconstructions. Interestingly, BADNeRF BADNeRF, combined with our embedding strategy, also provides clear results, being on par or slightly better than even when event data is used. The best results, however, are obtained with our method where, RGB images are used together with event data.