HDR Imaging for Dynamic Scenes with Events

TL;DR

This work tackles HDR imaging in real-world dynamic scenes affected by both saturated regions and motion blur. It presents Self-EHDRI, a self-supervised, event-based framework that jointly performs HDRI and motion deblurring by learning cross-domain mappings between blurry LDR and sharp LDR observations, guided by four self-consistency losses. The approach fuses multi-modal data through a Dynamic Range Enhancement module and a Motion Deblurring module, enabling sharp HDR outputs at arbitrary timestamps without ground-truth sharp HDR images. Comprehensive experiments on real-world and synthetic BL2SHD datasets demonstrate large improvements over state-of-the-art cascaded methods, and show practical gains in downstream tasks like object detection, highlighting the method’s impact for dynamic HDR imaging in uncontrolled environments.

Abstract

High dynamic range imaging (HDRI) for real-world dynamic scenes is challenging because moving objects may lead to hybrid degradation of low dynamic range and motion blur. Existing event-based approaches only focus on a separate task, while cascading HDRI and motion deblurring would lead to sub-optimal solutions, and unavailable ground-truth sharp HDR images aggravate the predicament. To address these challenges, we propose an Event-based HDRI framework within a Self-supervised learning paradigm, i.e., Self-EHDRI, which generalizes HDRI performance in real-world dynamic scenarios. Specifically, a self-supervised learning strategy is carried out by learning cross-domain conversions from blurry LDR images to sharp LDR images, which enables sharp HDR images to be accessible in the intermediate process even though ground-truth sharp HDR images are missing. Then, we formulate the event-based HDRI and motion deblurring model and conduct a unified network to recover the intermediate sharp HDR results, where both the high dynamic range and high temporal resolution of events are leveraged simultaneously for compensation. We construct large-scale synthetic and real-world datasets to evaluate the effectiveness of our method. Comprehensive experiments demonstrate that the proposed Self-EHDRI outperforms state-of-the-art approaches by a large margin. The codes, datasets, and results are available at https://lxp-whu.github.io/Self-EHDRI.

Figures (13)

• Figure 1: Previous paradigms vs. our paradigm. (a) Previous paradigms design cascading frameworks in a supervised manner, while ground-truth sharp HDR images can only be obtained in static scenes or dynamic scenes with specific dynamic attributes, e.g., actor pose, and controlled lighting conditions (stop-motion capture). When it comes to highly dynamic scenes that contain fast-moving targets, the ground-truth sharp HDR images are usually unavailable, leading to failure in real-world scenarios. (b) We propose an end-to-end and self-supervised framework by devising blurry LDR to sharp LDR conversion, which can handle highly dynamic scenes in the real world without ground-truth sharp HDR images.
• Figure 2: The illustration of our proposed self-supervised learning framework, i.e., Self-EHDRI. Top: overall pipeline of Self-EHDRI. The main branch comprises the E-BL2SH network, taking the blurry LDR image and concurrent events as input and generating sharp HDR images at arbitrary timestamps. The assistance branch contains DRD and DRC networks, performing dynamic range decomposition and composition to achieve flexible conversion between HDR and LDR domains. Bottom: the self-supervised consistencies, i.e., HDR-LDR consistency, LDR-LDR consistency, LDR-HDR consistency, and HDR-HDR consistency, enable the E-BL2SH task to be accomplished without ground-truth sharp HDR images.
• Figure 3: Structure of proposed Dynamic Range Decomposition (DRD) network and Dynamic Range Composition (DRC) network.
• Figure 4: Structure of proposed Event-guided Blurry LDR to Sharp HDR network, i.e., E-BL2SH, where event streams are embedded into voxel grids with polarity for processability.
• Figure 5: Samples from our proposed BL2SHD dataset, composed of three groups respectively for synthetic, i.e., BL2SHD-S, real-world static, i.e., BL2SHD-Rsimple, and real-world dynamic scenes i.e., BL2SHD-Rcomplex. Note that sharp HDR GT is absent for BL2SHD-Rcomplex.