Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation

Dimitrios Dagdilelis, Panagiotis Grigoriadis, Roberto Galeazzi

TL;DR

This work tackles robust BEV perception for autonomous marine navigation by fusing multi-modal sensors (RGB, LWIR, LiDAR via pseudo-views, X-band radar, ENC, and GNSS) within a cross-attention transformer framework. It introduces pseudo-camera LiDAR preprocessing, view-aware cross-attention with temporal aggregation, and end-to-end BEV supervision to produce a multi-class BEV map $\

Abstract

We propose a cross attention transformer based method for multimodal sensor fusion to build a birds eye view of a vessels surroundings supporting safer autonomous marine navigation. The model deeply fuses multiview RGB and long wave infrared images with sparse LiDAR point clouds. Training also integrates X band radar and electronic chart data to inform predictions. The resulting view provides a detailed reliable scene representation improving navigational accuracy and robustness. Real world sea trials confirm the methods effectiveness even in adverse weather and complex maritime settings.

Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation

TL;DR

This work tackles robust BEV perception for autonomous marine navigation by fusing multi-modal sensors (RGB, LWIR, LiDAR via pseudo-views, X-band radar, ENC, and GNSS) within a cross-attention transformer framework. It introduces pseudo-camera LiDAR preprocessing, view-aware cross-attention with temporal aggregation, and end-to-end BEV supervision to produce a multi-class BEV map $\

Abstract

We propose a cross attention transformer based method for multimodal sensor fusion to build a birds eye view of a vessels surroundings supporting safer autonomous marine navigation. The model deeply fuses multiview RGB and long wave infrared images with sparse LiDAR point clouds. Training also integrates X band radar and electronic chart data to inform predictions. The resulting view provides a detailed reliable scene representation improving navigational accuracy and robustness. Real world sea trials confirm the methods effectiveness even in adverse weather and complex maritime settings.
Paper Structure (16 sections, 16 equations, 14 figures, 4 tables)

This paper contains 16 sections, 16 equations, 14 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem formulation
  4. Task definition
  5. Existing methodologies
  6. Key problems
  7. Proposed Approach
  8. Dataset
  9. Solution architecture
  10. Motivation for Multi-modality
  11. Temporal features
  12. Experiments
  13. Implementation details
  14. Explainability
  15. Conclusion
  16. ...and 1 more sections

Figures (14)

  • Figure 1: High level overview of the proposed approach. Sensor streams are processed by modality-specific feature extractors. Modality features are processed as a sequence by the BEV Transformer model into a semantic BEV map. The final output representation is orthographic, compact and semantically useful for downstream navigation tasks.
  • Figure 2: Outlook from a virtual camera in a top down view pose, based on 360° images from four cameras. The scene is the entrance to a leisure craft harbor at Limfjorden (DK) \ref{['MartinsPaper']}
  • Figure 3: Tugboat Balder used for the data collection at Limfjorden, DK. Annotated are: a) XBR b) RGB & LWIR camera platform c) LiDAR
  • Figure 4: Ground truth BEV map generation. XBR data are transparently ploted below color-coded ENC classes (blue for land, magenta for shoreline, green for buoy, black for water). We curate moving target instances, by annotating magenta ellipsoids on top of verified moving targets.
  • Figure 5: Creating a ground truth BEV map $y$ using XBR, ENC, satellite-compass, and GNSS.
  • ...and 9 more figures