Temporal-Mapping Photography for Event Cameras

TL;DR

For the first time, for the first time, it is realized that events to dense intensity image conversion using a stationary event camera in static scenes with a transmittance adjustment device for brightness modulation is realized.

Abstract

Event cameras, or Dynamic Vision Sensors (DVS) are novel neuromorphic sensors that capture brightness changes as a continuous stream of "events" rather than traditional intensity frames. Converting sparse events to dense intensity frames faithfully has long been an ill-posed problem. Previous methods have primarily focused on converting events to video in dynamic scenes or with a moving camera. In this paper, for the first time, we realize events to dense intensity image conversion using a stationary event camera in static scenes with a transmittance adjustment device for brightness modulation. Different from traditional methods that mainly rely on event integration, the proposed Event-Based Temporal Mapping Photography (EvTemMap) measures the time of event emitting for each pixel. Then, the resulting Temporal Matrix is converted to an intensity frame with a temporal mapping neural network. At the hardware level, the proposed EvTemMap is implemented by combining a transmittance adjustment device with a DVS, named Adjustable Transmittance Dynamic Vision Sensor (AT-DVS). Additionally, we collected TemMat dataset under various conditions including low-light and high dynamic range scenes. The experimental results showcase the high dynamic range, fine-grained details, and high-grayscale resolution of the proposed EvTemMap. The code and dataset are available in https://github.com/YuHanBaozju/EvTemMap

Figures (8)

• Figure 1: Pipeline of EvTemMap. The actual scenes are initially recorded by AT-DVS, generating a Temporal Matrix. Following this, the Temporal Matrix undergoes time-intensity mapping and denoising through a Temporal-mapping network, producing a grayscale image.
• Figure 2: An implementation example of EvTemMap. (a) Temporal Matrix. (b)The intensity image obtained by substituting Temporal Matrix into \ref{['t2image']}. (c) The adaptive dynamic range image ensures optimal grayscale resolution without underexposure or overexposure. (d) Restricted dynamic range image shows overexposing and underexposing.
• Figure 3: Degradation paradigm. (a) The real IPE timestamp is affected by the threshold random perturbation $\Delta V_{thd}$ and the time measurement error $\delta t$. (b) The hot pixel is reflected as an unusually early value on the Temporal Matrix and as the brightest white dot in the grayscale image. (c) The process of generating training data for the degradation model.
• Figure 4: Qualitative results on TemMat dataset. Rows from top to bottom: Visualized event frame, results of E2VID rebecq2019events and E2VID+ stoffregen2020reducing, visualized Temporal Matrix, results of EvTemMap, reference image. Our method achieves the most faithful and finest result. Best viewed on a screen and zoomed in.
• Figure 5: Comparison in high dynamic range scenes. The brightest and darkest areas of (a) and (c) are rendered in pseudo-color. Overexposed areas and underexposed areas in (c) are marked in red and blue, respectively. EvTemMap presents exceptional HDR photography with minimal noise. Conversely, E2VID exhibits a loss of many texture features, while Conventional LDR suffers from inadequate information capture in both underexposed and overexposed regions.