Event-based Motion Deblurring via Multi-Temporal Granularity Fusion

TL;DR

The paper tackles motion blur in frame-based cameras by leveraging high-temporal-resolution event data. It introduces MTGNet, a dual-branch architecture that fuses a voxel-based coarse-temporal representation with a fine-grained point-cloud representation via an Aggregation and Mapping Module (AMM) and an Adaptive Feature Diffusion Module (AFDM). The method links blur formation to event streams through a discretized time formulation and optimizes a composite loss combining $L_{MAE}$, $L_{SSIM}$, and $L_{MSFR}$, demonstrating state-of-the-art performance on Ev-REDS, HS-ERGB, and MS-RBD with thorough ablations validating multi-temporal granularity and diffusion. The findings highlight the practical impact of aligning and diffusing sparse, high-temporal-resolution event data with frame data to robustly restore sharp images in challenging dynamic scenes.

Abstract

Conventional frame-based cameras inevitably produce blurry effects due to motion occurring during the exposure time. Event camera, a bio-inspired sensor offering continuous visual information could enhance the deblurring performance. Effectively utilizing the high-temporal-resolution event data is crucial for extracting precise motion information and enhancing deblurring performance. However, existing event-based image deblurring methods usually utilize voxel-based event representations, losing the fine-grained temporal details that are mathematically essential for fast motion deblurring. In this paper, we first introduce point cloud-based event representation into the image deblurring task and propose a Multi-Temporal Granularity Network (MTGNet). It combines the spatially dense but temporally coarse-grained voxel-based event representation and the temporally fine-grained but spatially sparse point cloud-based event. To seamlessly integrate such complementary representations, we design a Fine-grained Point Branch. An Aggregation and Mapping Module (AMM) is proposed to align the low-level point-based features with frame-based features and an Adaptive Feature Diffusion Module (AFDM) is designed to manage the resolution discrepancies between event data and image data by enriching the sparse point feature. Extensive subjective and objective evaluations demonstrate that our method outperforms current state-of-the-art approaches on both synthetic and real-world datasets.

Figures (8)

• Figure 1: Comparison of deblurring results with the state-of-the-art event-based motion deblurring methods, including TRMD, GEM, and our MTGNet.
• Figure 2: The overall framework of MTGNet. The event stream is shaped into the point cloud-based representation $\mathcal{P}$ and the voxel-based representation $\mathcal{V}$. $I_{i}$ are the multi-scale blurry images. MTGNet consists of a Coarse Temporal Fusion Branch and a Fine-grained Point Branch. AMM is the Aggregation and Mapping Module, AFDM is the Adaptive Feature Diffusion Module.
• Figure 3: Qualitative comparisons under Ev-REDS dataset. Best viewed on a screen and zoomed in .
• Figure 4: Qualitative comparisons under Ev-REDS dataset. Best viewed on a screen and zoomed in .
• Figure 5: Qualitative comparisons under HS-ERGB dataset. Best viewed on a screen and zoomed in.