Seeing through Light and Darkness: Sensor-Physics Grounded Deblurring HDR NeRF from Single-Exposure Images and Events

TL;DR

This work tackles HDR novel view synthesis from a single-exposure blurry LDR image paired with events, addressing the sensor-physics mismatch that hampers prior ERGB methods. It introduces See-NeRF, a sensor-physics grounded NeRF framework that models HDR radiance with a pixel-wise RGB mapping field and a latency-aware, photometrically calibrated event mapping field, all optimized jointly with the NeRF to produce sharp HDR 3D representations. See-NeRF leverages volume rendering to simulate HDR scene rays, applies a pixel-level CRF-aware tone mapping, and uses event dynamics to recover scene motion while compensating for sensor delays and photometric nonlinearities; the loss combines LDR and event supervision. Experiments on synthetic and real datasets show state-of-the-art performance for both HDR NVS and deblurring NVS, validating the physical grounding and the complementary use of events for improved sharpness and dynamic-range in 3D reconstructions.

Abstract

Novel view synthesis from low dynamic range (LDR) blurry images, which are common in the wild, struggles to recover high dynamic range (HDR) and sharp 3D representations in extreme lighting conditions. Although existing methods employ event data to address this issue, they ignore the sensor-physics mismatches between the camera output and physical world radiance, resulting in suboptimal HDR and deblurring results. To cope with this problem, we propose a unified sensor-physics grounded NeRF framework for sharp HDR novel view synthesis from single-exposure blurry LDR images and corresponding events. We employ NeRF to directly represent the actual radiance of the 3D scene in the HDR domain and model raw HDR scene rays hitting the sensor pixels as in the physical world. A pixel-wise RGB mapping field is introduced to align the above rendered pixel values with the sensor-recorded LDR pixel values of the input images. A novel event mapping field is also designed to bridge the physical scene dynamics and actual event sensor output. The two mapping fields are jointly optimized with the NeRF network, leveraging the spatial and temporal dynamic information in events to enhance the sharp HDR 3D representation learning. Experiments on the collected and public datasets demonstrate that our method can achieve state-of-the-art deblurring HDR novel view synthesis results with single-exposure blurry LDR images and corresponding events.

Figures (14)

• Figure 1: Events corresponding to an LDR blurry image contain spatial difference (High Dynamic Range) and temporal difference (High Temporal Resolution) information of the scene radiance, making it an ideal data for enhancing HDR sharp 3D scene reconstruction. See-NeRF models the physical imaging process with proposed RGB and event mapping fields to bridge the sensor-physics discrepancy, leveraging events to infer scene dynamics and achieving sharper and better HDR novel view synthesis results compared to the physical unaligned method EvHDR-NeRF.
• Figure 2: HDR reconstruction from Events. The first row shows the input data. The second row shows the estimated CRF curves and the dynamic range schematic. For a single-exposure LDR image with exposure time $\Delta t_2$, its 8-bit pixel values 0-255 (normalized to 0-1 as $\mathcal{I}_{LDR}$ in the figure) represent a limited range of scene radiance $\ln(L(t))$ (width of the blue box). While traditional methods fuse multi-exposure LDR images with exposure times $\Delta t_0, \Delta t_2, \Delta t_4$ to compensate for the limited scene brightness representation. We leverage the spatial differential in events to infer the scene radiance of low-light and highlight regions from LDR image ($\Delta t_2$), which expands the representation of scene radiance, enabling more accurate CRF estimation curves in \ref{['sec:4.2']}.
• Figure 3: The data generation of event and RGB cameras (upper parts) and the pipeline of See-NeRF (lower parts) in an extreme lighting scene. We use NeRF to represent the actual radiance of the scene in the HDR domain. The volume rendering simulates the raw scene radiance rays hitting the sensor physically and obtaining the raw sensor values. The predicted LDR images and events are generated with our proposed RGB and event mapping fields, built upon the data generation model of the sensors. The image loss and event loss are employed to jointly supervise the optimization of the two mapping fields and the NeRF network.
• Figure 4: Effectiveness of our RGB mapping field.
• Figure 5: Qualitative results on the synthetic "dogroom" scene and real-world "bear" scene of our datasets for the HDR novel view synthesis task. Our See-NeRF achieves the best results on both under-exposure and over-exposure regions.