Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Line Segments in Motion-blurred Images with Events

Huai Yu, Hao Li, Wen Yang, Lei Yu, Gui-Song Xia

TL;DR

This work designs a general frame-event feature fusion network to extract and fuse the detailed image textures and low-latency event edges and utilizes the state-of-the-art wireframe parsing networks to detect line segments on the fused feature map.

Abstract

Making line segment detectors more reliable under motion blurs is one of the most important challenges for practical applications, such as visual SLAM and 3D reconstruction. Existing line segment detection methods face severe performance degradation for accurately detecting and locating line segments when motion blur occurs. While event data shows strong complementary characteristics to images for minimal blur and edge awareness at high-temporal resolution, potentially beneficial for reliable line segment recognition. To robustly detect line segments over motion blurs, we propose to leverage the complementary information of images and events. To achieve this, we first design a general frame-event feature fusion network to extract and fuse the detailed image textures and low-latency event edges, which consists of a channel-attention-based shallow fusion module and a self-attention-based dual hourglass module. We then utilize two state-of-the-art wireframe parsing networks to detect line segments on the fused feature map. Besides, we contribute a synthetic and a realistic dataset for line segment detection, i.e., FE-Wireframe and FE-Blurframe, with pairwise motion-blurred images and events. Extensive experiments on both datasets demonstrate the effectiveness of the proposed method. When tested on the real dataset, our method achieves 63.3% mean structural average precision (msAP) with the model pre-trained on the FE-Wireframe and fine-tuned on the FE-Blurframe, improved by 32.6 and 11.3 points compared with models trained on synthetic only and real only, respectively. The codes, datasets, and trained models are released at: https://levenberg.github.io/FE-LSD

Detecting Line Segments in Motion-blurred Images with Events

TL;DR

This work designs a general frame-event feature fusion network to extract and fuse the detailed image textures and low-latency event edges and utilizes the state-of-the-art wireframe parsing networks to detect line segments on the fused feature map.

Abstract

Making line segment detectors more reliable under motion blurs is one of the most important challenges for practical applications, such as visual SLAM and 3D reconstruction. Existing line segment detection methods face severe performance degradation for accurately detecting and locating line segments when motion blur occurs. While event data shows strong complementary characteristics to images for minimal blur and edge awareness at high-temporal resolution, potentially beneficial for reliable line segment recognition. To robustly detect line segments over motion blurs, we propose to leverage the complementary information of images and events. To achieve this, we first design a general frame-event feature fusion network to extract and fuse the detailed image textures and low-latency event edges, which consists of a channel-attention-based shallow fusion module and a self-attention-based dual hourglass module. We then utilize two state-of-the-art wireframe parsing networks to detect line segments on the fused feature map. Besides, we contribute a synthetic and a realistic dataset for line segment detection, i.e., FE-Wireframe and FE-Blurframe, with pairwise motion-blurred images and events. Extensive experiments on both datasets demonstrate the effectiveness of the proposed method. When tested on the real dataset, our method achieves 63.3% mean structural average precision (msAP) with the model pre-trained on the FE-Wireframe and fine-tuned on the FE-Blurframe, improved by 32.6 and 11.3 points compared with models trained on synthetic only and real only, respectively. The codes, datasets, and trained models are released at: https://levenberg.github.io/FE-LSD
Paper Structure (24 sections, 9 equations, 15 figures, 6 tables)

This paper contains 24 sections, 9 equations, 15 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image-based line segment detection
  4. Event-based line segment detection
  5. Methodology
  6. Problem statement
  7. Event representation
  8. Network structure
  9. Shallow module
  10. Dual hourglass module
  11. Line Segment Detector
  12. Experiments and Analysis
  13. Dataset
  14. Experimental settings and evaluation metrics
  15. Configuration analyses
  16. ...and 9 more sections

Figures (15)

  • Figure 1: Illustrative examples of different methods. Line segments are drawn on clear images at the end of camera exposure for better visualization. Conventional (d)LSD von2010lsd and (e)FBSD even2019thick have many false detections while the original (f)LETR xu_line_2021, (g)HAWP xue2020holistically and (j)ULSD li2021ulsd have many missing alarms on the (a)motion-blurred image. The retrained (h)HAWP and (k)ULSD on the concatenation of (a)images and (b)events detect more line segments. The proposed (i)FE-HAWP and (l)FE-ULSD have the best performance.
  • Figure 2: Blurred line segments in motion-blurred images. From left to right are the line segments at the beginning, end, and within the camera exposure time.
  • Figure 3: Event data representation using the EST (Left: raw event stream, red-positive events, blue-negative events; Right: EST representation with $B=5$).
  • Figure 4: FE-LSD network structure. It mainly consists of two modules: feature fusion backbone and back-end line detector, in which the feature fusion backbone has one shallow module and several dual hourglass modules.
  • Figure 5: Shallow Fusion Block (SFB).
  • ...and 10 more figures