DAVIDE: Depth-Aware Video Deblurring

TL;DR

Video deblurring benefits from depth information, but the advantage is context-dependent. The authors introduce the DAVIDE RGB-D video deblurring dataset and develop a depth-aware extension of Shift-Net, incorporating a Depth-aware Transformer Block (DaT) and Depth fusion through GSTS and GSS mechanisms. Experimental results show depth cues boost performance for small temporal contexts, with gains diminishing as the context window increases, while depth reliability and scene attributes modulate the benefits. The work provides practical insights for leveraging depth sensors in real-world video restoration and establishes a valuable benchmark for depth-aware video deblurring research.

Abstract

Video deblurring aims at recovering sharp details from a sequence of blurry frames. Despite the proliferation of depth sensors in mobile phones and the potential of depth information to guide deblurring, depth-aware deblurring has received only limited attention. In this work, we introduce the 'Depth-Aware VIdeo DEblurring' (DAVIDE) dataset to study the impact of depth information in video deblurring. The dataset comprises synchronized blurred, sharp, and depth videos. We investigate how the depth information should be injected into the existing deep RGB video deblurring models, and propose a strong baseline for depth-aware video deblurring. Our findings reveal the significance of depth information in video deblurring and provide insights into the use cases where depth cues are beneficial. In addition, our results demonstrate that while the depth improves deblurring performance, this effect diminishes when models are provided with a longer temporal context. Project page: https://germanftv.github.io/DAVIDE.github.io/ .

Figures (10)

• Figure 1: Examples of depth-aware video deblurring (RGBD) with increasing temporal length $T$ of the context window (see \ref{['sec:exp_depth_impact']} for details).
• Figure 2: The DAVIDE blur synthesis pipeline.
• Figure 3: Overview of Shift-Net.
• Figure 4: Depth fusion block.
• Figure 5: Impact of the depth cue with varying $T$ (temporal length of the context window); (a) 95% confidence ($\pm$2 std) plot between the RGB and RGBD performance; (b) avg. results over three independent runs.