ADV2E: Bridging the Gap Between Analogue Circuit and Discrete Frames in the Video-to-Events Simulator

TL;DR

Experimental results on two relevant tasks, including semantic segmentation and image reconstruction, validate the reliability of simulated event data, even in high-contrast scenes, confirming that the synthetic events generated by the proposed method are both realistic and well-suited for effective training.

Abstract

Event cameras operate fundamentally differently from traditional Active Pixel Sensor (APS) cameras, offering significant advantages. Recent research has developed simulators to convert video frames into events, addressing the shortage of real event datasets. Current simulators primarily focus on the logical behavior of event cameras. However, the fundamental analogue properties of pixel circuits are seldom considered in simulator design. The gap between analogue pixel circuit and discrete video frames causes the degeneration of synthetic events, particularly in high-contrast scenes. In this paper, we propose a novel method of generating reliable event data based on a detailed analysis of the pixel circuitry in event cameras. We incorporate the analogue properties of event camera pixel circuits into the simulator design: (1) analogue filtering of signals from light intensity to events, and (2) a cutoff frequency that is independent of video frame rate. Experimental results on two relevant tasks, including semantic segmentation and image reconstruction, validate the reliability of simulated event data, even in high-contrast scenes. This demonstrates that deep neural networks exhibit strong generalization from simulated to real event data, confirming that the synthetic events generated by the proposed method are both realistic and well-suited for effective training.

Figures (6)

• Figure 1: The framework of the proposed ADV2E simulator. Different from v2e, the proposed simulator models brightness changes by faithfully emulating the core analogue behaviors inherent to real DVS circuitry, including continuity sampling and analogue low-pass filtering. Therefore, the proposed event simulator addresses the issue of inaccurate filtering delays in current simulators, improving the accuracy of simulated events.
• Figure 2: Continuity Sampling. The frame rate is significantly increased through linear over-sampling, effectively preventing aliasing and preserving the analogue behaviors of DVS.
• Figure 3: Analogue low-pass filtering. In a real DVS circuit, the cutoff frequency of the low-pass filters varies over time, proportional to the current brightness. By applying continuous brightness values, the proposed simulator closely approximates the behaviors of DVS analogue circuit, enabling realistic event generation.
• Figure 4: Qualitative comparison of events generated by different simulators. Details are best viewed when zoomed in. Each column represents a frame from the DAVIS dataset Mueggler2017. From top to bottom, there are (a) APS frames, events synthesized by (b) vid2e Rebecq2018Gehrig2020, (c) v2e Hu2021, (d) v2ce Zhang2024, (e) DVS-Voltmeter Lin2022, (f) the proposed ADV2E, and (g) ground truth. The events produced by ADV2E most closely resemble the ground truth, with realistic event generation in high-contrast regions, such as the balconies and the intersections between buildings and roads.
• Figure 5: Qualitative comparison of semantic segmentation across event simulators. Each column represents a separate scene. From top to bottom, the rows show (a) APS frames, (b) events, segmentation results by (c) vid2e Rebecq2018Gehrig2020, (d) v2e Hu2021, (e) v2ce Zhang2024, (f) DVS-Voltmeter Lin2022, (g) the proposed ADV2E, and (h) ground truth. The ground truth is generated automatically by a CNN Alonso2019 on APS images. Compared to other simulators, the events generated by the proposed ADV2E simulator enable the CNN to capture objects with greater accuracy. For example, the pedestrian in the 1st column, the bridge in the 2nd column, and all vehicles are segmented precisely by the proposed ADV2E. In the 5th column, only the network trained on events from ADV2E correctly identifies the traffic light on the left. Overall, ADV2E produces the most realistic events among the simulators evaluated.