EventHDR: from Event to High-Speed HDR Videos and Beyond

TL;DR

This paper presents a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data.

Abstract

Event cameras are innovative neuromorphic sensors that asynchronously capture the scene dynamics. Due to the event-triggering mechanism, such cameras record event streams with much shorter response latency and higher intensity sensitivity compared to conventional cameras. On the basis of these features, previous works have attempted to reconstruct high dynamic range (HDR) videos from events, but have either suffered from unrealistic artifacts or failed to provide sufficiently high frame rates. In this paper, we present a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data. Additionally, to address the problem of severely limited real dataset, we develop a new optical system to collect a real-world dataset with paired high-speed HDR videos and event streams, facilitating future research in this field. Our dataset provides the first real paired dataset for event-to-HDR reconstruction, avoiding potential inaccuracies from simulation strategies. Experimental results demonstrate that our method can generate high-quality, high-speed HDR videos. We further explore the potential of our work in cross-camera reconstruction and downstream computer vision tasks, including object detection, panoramic segmentation, optical flow estimation, and monocular depth estimation under HDR scenarios.

Figures (15)

• Figure 1: The comparison between previous methods/APS and our method. Previous work rebecq2019high lacks real high bit-depth HDR images as ground truth, resulting in reconstructed images that suffer from severe artifacts. In contrast, APS images exhibit a low dynamic range, leading to suboptimal performance in downstream vision tasks under HDR scenes. Our method, however, produces visually pleasing results in both HDR reconstruction and real HDR applications, showing its superiority over existing approaches.
• Figure 2: The overview of our recurrent convolutional neural network for HDR video reconstruction from events.
• Figure 3: The hardware implementation of our high-speed HDR video imaging system with events. (a) The optical system. It contains two high-speed cameras and an event camera. (b) The electronic system. It controls three cameras to synchronously capture the information of same scene. (c) The synchronize data captured by three cameras, respectively.
• Figure 4: Three representative scenes of our captured real dataset. In order to recognize the scene motions, the shown two consecutive frames are chosen at the interval of $100$ real frames.
• Figure 5: Qualitative reconstruction results on simulated data.