Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RGB-Sonar Tracking Benchmark and Spatial Cross-Attention Transformer Tracker

Yunfeng Li, Bo Wang, Jiuran Sun, Xueyi Wu, Ye Li

TL;DR

Comprehensive experiments show that the proposed spatial cross-attention effectively achieves the interaction between RGB and sonar modalities, and that SCANet and SCANet-Refine achieves state-of-the-art performance on the proposed benchmark.

Abstract

Vision camera and sonar are naturally complementary in the underwater environment. Combining the information from two modalities will promote better observation of underwater targets. However, this problem has not received sufficient attention in previous research. Therefore, this paper introduces a new challenging RGB-Sonar (RGB-S) tracking task and investigates how to achieve efficient tracking of an underwater target through the interaction of RGB and sonar modalities. Specifically, we first propose an RGBS50 benchmark dataset containing 50 sequences and more than 87000 high-quality annotated bounding boxes. Experimental results show that the RGBS50 benchmark poses a challenge to currently popular SOT trackers. Second, we propose an RGB-S tracker called SCANet, which includes a spatial cross-attention module (SCAM) consisting of a novel spatial cross-attention layer and two independent global integration modules. The spatial cross-attention is used to overcome the problem of spatial misalignment of between RGB and sonar images. Third, we propose a SOT data-based RGB-S simulation training method (SRST) to overcome the lack of RGB-S training datasets. It converts RGB images into sonar-like saliency images to construct pseudo-data pairs, enabling the model to learn the semantic structure of RGB-S-like data. Comprehensive experiments show that the proposed spatial cross-attention effectively achieves the interaction between RGB and sonar modalities and SCANet achieves state-of-the-art performance on the proposed benchmark. The code is available at https://github.com/LiYunfengLYF/RGBS50.

RGB-Sonar Tracking Benchmark and Spatial Cross-Attention Transformer Tracker

TL;DR

Comprehensive experiments show that the proposed spatial cross-attention effectively achieves the interaction between RGB and sonar modalities, and that SCANet and SCANet-Refine achieves state-of-the-art performance on the proposed benchmark.

Abstract

Vision camera and sonar are naturally complementary in the underwater environment. Combining the information from two modalities will promote better observation of underwater targets. However, this problem has not received sufficient attention in previous research. Therefore, this paper introduces a new challenging RGB-Sonar (RGB-S) tracking task and investigates how to achieve efficient tracking of an underwater target through the interaction of RGB and sonar modalities. Specifically, we first propose an RGBS50 benchmark dataset containing 50 sequences and more than 87000 high-quality annotated bounding boxes. Experimental results show that the RGBS50 benchmark poses a challenge to currently popular SOT trackers. Second, we propose an RGB-S tracker called SCANet, which includes a spatial cross-attention module (SCAM) consisting of a novel spatial cross-attention layer and two independent global integration modules. The spatial cross-attention is used to overcome the problem of spatial misalignment of between RGB and sonar images. Third, we propose a SOT data-based RGB-S simulation training method (SRST) to overcome the lack of RGB-S training datasets. It converts RGB images into sonar-like saliency images to construct pseudo-data pairs, enabling the model to learn the semantic structure of RGB-S-like data. Comprehensive experiments show that the proposed spatial cross-attention effectively achieves the interaction between RGB and sonar modalities and SCANet achieves state-of-the-art performance on the proposed benchmark. The code is available at https://github.com/LiYunfengLYF/RGBS50.
Paper Structure (28 sections, 4 equations, 11 figures, 8 tables)

This paper contains 28 sections, 4 equations, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Single Object Tracking
  4. Underwater Tracking
  5. Multi-Modal Tracking
  6. RGB-Sonar Tracking Benchmark
  7. RGBS50 Benchmark
  8. Baseline Methods
  9. Evaluation Metrics
  10. Method
  11. Backbone
  12. Spatial Cross-Attention Module
  13. Head and Loss
  14. SOT data-based training
  15. experiments
  16. ...and 13 more sections

Figures (11)

  • Figure 1: Comparison of the RGB-S tracking task with other multimodal tracking tasks. The RGB images and sonar images have spatial misalignment characteristic. (a) Paired RGB-T images of LasHeR lasher, (b) Paired RGB-D images of DepthTrack depthtrack, (c) Paired RGB-E images of VOT2019-RGBE vot2019rgbe, (d) Paired RGB-S images of our proposed dataset.
  • Figure 2: Main features of the proposed dataset. (a) The multi-sensor platform we use for data collection. (b) Quantity distribution of 7 object categories (name and percentage). (c) Statistics on frame-level and sequence-level attributes.
  • Figure 3: Sample RGB and sonar sequences with annotated attributes from our RGBS50 dataset. The black font represents the sequence name. The blue font represents the annotated attributes.
  • Figure 4: The overall framework of SCANet. We take the ViT pre-trained by SOT ostrack as our baseline, and insert the proposed SCAM module into different layers in the backbone to achieve multi-level cross-modal feature interactions. Finally, the output features of each branch are fed into the corresponding prediction head to predict the state of the target in the two modalities images, respectively.
  • Figure 5: Illustration of the Sonar to RGB interaction process of our proposed spatial cross-attention. The process of RGB to Sonar is the same.
  • ...and 6 more figures