Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization

Ziran Zhang, Yongrui Ma, Yueting Chen, Feng Zhang, Jinwei Gu, Tianfan Xue, Shi Guo

TL;DR

This work proposes a novel per-scene optimization strategy that utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings.

Abstract

Video Frame Interpolation (VFI) is important for video enhancement, frame rate up-conversion, and slow-motion generation. The introduction of event cameras, which capture per-pixel brightness changes asynchronously, has significantly enhanced VFI capabilities, particularly for high-speed, nonlinear motions. However, these event-based methods encounter challenges in low-light conditions, notably trailing artifacts and signal latency, which hinder their direct applicability and generalization. Addressing these issues, we propose a novel per-scene optimization strategy tailored for low-light conditions. This approach utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings. To evaluate its robustness in low-light condition, we further introduce EVFI-LL, a unique RGB+Event dataset captured under low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments. Project page: https://naturezhanghn.github.io/sim2real.

From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization

TL;DR

This work proposes a novel per-scene optimization strategy that utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings.

Abstract

Video Frame Interpolation (VFI) is important for video enhancement, frame rate up-conversion, and slow-motion generation. The introduction of event cameras, which capture per-pixel brightness changes asynchronously, has significantly enhanced VFI capabilities, particularly for high-speed, nonlinear motions. However, these event-based methods encounter challenges in low-light conditions, notably trailing artifacts and signal latency, which hinder their direct applicability and generalization. Addressing these issues, we propose a novel per-scene optimization strategy tailored for low-light conditions. This approach utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings. To evaluate its robustness in low-light condition, we further introduce EVFI-LL, a unique RGB+Event dataset captured under low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments. Project page: https://naturezhanghn.github.io/sim2real.
Paper Structure (24 sections, 11 equations, 8 figures, 4 tables)

This paper contains 24 sections, 11 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RGB-based Video Frame Interpolation
  4. Event-based Video Frame Interpolation
  5. Low-light Event Correction
  6. Problem definition
  7. Low-light Event Representation
  8. Low Light Event-VFI
  9. Method
  10. Per-scene Optimization for VFI
  11. Optimization Loss
  12. Low Light Event Simulation
  13. Experiments
  14. Experiment Details
  15. Low Light Event-VFI Benchmark: EVFI-LL
  16. ...and 9 more sections

Figures (8)

  • Figure 1: Illustration of the proposed per-scene optimization process.
  • Figure 2: Overview of the proposed low light Event-VFI method. This framework contains the pre-training phase, per-scene optimization, and inference process.
  • Figure 3: Simplified circuit diagram for each pixel of the event camera. And the visualization of event trailing artifacts, captured by our EVS-RGB beam splitting imaging system.
  • Figure 4: Synthesis process for the EVS Motion Trail Simulation Dataset.
  • Figure 5: Visual comparison of the low-light trailing simulation results of the event camera with v2e.
  • ...and 3 more figures