Deep Lidar-guided Image Deblurring

TL;DR

The paper tackles image deblurring in low-light smartphone imaging by leveraging true depth from mobile LiDAR as auxiliary information. It introduces a universal depth adapter, depth-super-resolution, and a continual-learning strategy to integrate depth into pretrained deblurring models, exemplified by Depth-Restormer. Empirical results show substantial PSNR/SSIM gains across several architectures, with real LiDAR depth outperforming depth estimated from blurry images and 8× SR depth approaching high-resolution LiDAR performance. The approach provides a practical, modular path to depth-aware deblurring, improving edge preservation and restoration quality, while acknowledging limitations such as texture absence and depth-range constraints.

Abstract

The rise of portable Lidar instruments, including their adoption in smartphones, opens the door to novel computational imaging techniques. Being an active sensing instrument, Lidar can provide complementary data to passive optical sensors, particularly in situations like low-light imaging where motion blur can affect photos. In this paper, we study if the depth information provided by mobile Lidar sensors is useful for the task of image deblurring and how to integrate it with a general approach that transforms any state-of-the-art neural deblurring model into a depth-aware one. To achieve this, we developed a universal adapter structure that efficiently preprocesses the depth information to modulate image features with depth features. Additionally, we applied a continual learning strategy to pretrained encoder-decoder models, enabling them to incorporate depth information as an additional input with minimal extra data requirements. We demonstrate that utilizing true depth information can significantly boost the effectiveness of deblurring algorithms, as validated on a dataset with real-world depth data captured by a smartphone Lidar.

Figures (7)

• Figure 1: Pipeline of a generic depth-guided image deblurring model. The encoder part of the deblurring pretrained model is frozen, while the decoder part is trainable. Depth feature extraction is combined with super-resolution, if needed. Adapters modulate image features in the decoder model according to depth features.
• Figure 2: The depth-restormer architecture example. The adapter is added on each level of the decoder part, which is in front of the transformer block. There are also 3 depth blocks corresponding to each adapter. The architecture of the depth block and the adapter is also shown in the figure.
• Figure 3: Depth super-resolution architecture.
• Figure 4: Left to right: Mobile Lidar depth map, Restormer deblurred image, Depth-Restormer, ground truth. As shown in the detail, Depth-Restormer has sharper object edges.
• Figure 5: Visual comparison of deblurring results for the considered state-of-the-art models. For each scene, the top row presents results of the original model, while the bottom row presents results of the depth-enhanced model.