RED: Robust Event-Guided Motion Deblurring with Modality-Specific Disentanglement

TL;DR

A Robustness-Oriented Perturbation Strategy (RPS) is introduced that mimics various trigger thresholds of dynamic vision sensors, exposing the model to diverse under-reporting patterns and thereby improving robustness under unknown conditions.

Abstract

Event-guided motion deblurring reconstructs sharp images using the high-temporal-resolution motion cues from event cameras. However, in real capture, thresholding-induced event under-reporting causes missing and fragmented motion cues, under which existing methods often degrade in performance due to two limitations: i) assumptions of dense and stable events, and ii) modality-indiscriminate extraction and fusion that fail to separate useful motion cues from disrupted events, allowing them to contaminate cross-modal representations. In this paper, we first introduce a Robustness-Oriented Perturbation Strategy (RPS) that mimics various trigger thresholds of dynamic vision sensors, exposing our model to diverse under-reporting patterns and thereby improving robustness under unknown conditions. Built upon this setting, we propose RED, a Robust Event-guided Deblurring network, following the principle of disentangle first and then fuse selectively. Specifically, the Modality-specific Representation Mechanism disentangles the inputs into image-semantic, event-motion, and cross-modal representations, capturing appearance, motion, and complementary interactions, respectively. With the reliable disentangled features, we selectively fuse modalities to enhance motion-sensitive areas in blurry images and enrich under-reported events with semantic context. Extensive experiments on synthetic and real-world datasets demonstrate RED consistently achieves state-of-the-art performance in terms of both accuracy and robustness.

Figures (9)

• Figure 1: Threshold-driven under-reporting: weak motions that fall below the trigger condition are not reported.
• Figure 2: As the under-reporting ratio increases, the performances of existing event-based deblurring methods degrade sharply and fall below the image-only method DSTN.
• Figure 3: Overview of our RED. In detail, a Robustness-Oriented Perturbation Strategy (RPS) is implemented to event input, and MRM is designed to disentangle modality-specific features with individually semantic reasoning in image module, motion-wise representation in motion module, and cross-modality fusion in cross-modal module. Furthermore, Motion Saliency Enhancer Module (MSEM) is designed to excavate motion-sensitive priors to image branch and Event Semantic Engraver Module (ESEM) is presented to compensate events with global semantic understanding.
• Figure 4: Framework of our proposed Motion Saliency Enhancer Module (MSEM) and Event Semantic Engraver Module (ESEM).
• Figure 5: Visualizations of the activation maps, including: 1) image-encoder features before and after MSEM, 2) image-branch decoder features, 3) event-encoder features before and after ESEM, and 4) event-branch decoder features.