Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset

Shijie Lian, Ziyi Zhang, Hua Li, Wenjie Li, Laurence Tianruo Yang, Sam Kwong, Runmin Cong

TL;DR

This work tackles underwater salient instance segmentation by introducing the USIS10K dataset (10,632 images across 7 categories) and a SAM-guided architecture, USIS-SAM, that incorporates a domain-aware Underwater Adaptive ViT (UA-ViT) encoder and an automatic Salient Feature Prompter Generator (SFPG). The UA-ViT injects underwater prompts through adapters and a Channel Adapter, while SFPG fuses multi-scale features to produce prompts for SAM, enabling end-to-end segmentation without manual prompts. Extensive experiments on USIS10K show that USIS-SAM achieves state-of-the-art performance in both class-agnostic and multi-class SIS, with ablations confirming the contributions of UA-ViT, CA, and SFPG; transferability to related underwater tasks and a land dataset (SIS10K) is also explored. The dataset and code are released to support research in underwater vision and SAM-based segmentation under challenging environmental conditions, potentially benefiting marine exploration and robotic underwater perception.

Abstract

With the breakthrough of large models, Segment Anything Model (SAM) and its extensions have been attempted to apply in diverse tasks of computer vision. Underwater salient instance segmentation is a foundational and vital step for various underwater vision tasks, which often suffer from low segmentation accuracy due to the complex underwater circumstances and the adaptive ability of models. Moreover, the lack of large-scale datasets with pixel-level salient instance annotations has impeded the development of machine learning techniques in this field. To address these issues, we construct the first large-scale underwater salient instance segmentation dataset (USIS10K), which contains 10,632 underwater images with pixel-level annotations in 7 categories from various underwater scenes. Then, we propose an Underwater Salient Instance Segmentation architecture based on Segment Anything Model (USIS-SAM) specifically for the underwater domain. We devise an Underwater Adaptive Visual Transformer (UA-ViT) encoder to incorporate underwater domain visual prompts into the segmentation network. We further design an out-of-the-box underwater Salient Feature Prompter Generator (SFPG) to automatically generate salient prompters instead of explicitly providing foreground points or boxes as prompts in SAM. Comprehensive experimental results show that our USIS-SAM method can achieve superior performance on USIS10K datasets compared to the state-of-the-art methods. Datasets and codes are released on https://github.com/LiamLian0727/USIS10K.

Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset

TL;DR

This work tackles underwater salient instance segmentation by introducing the USIS10K dataset (10,632 images across 7 categories) and a SAM-guided architecture, USIS-SAM, that incorporates a domain-aware Underwater Adaptive ViT (UA-ViT) encoder and an automatic Salient Feature Prompter Generator (SFPG). The UA-ViT injects underwater prompts through adapters and a Channel Adapter, while SFPG fuses multi-scale features to produce prompts for SAM, enabling end-to-end segmentation without manual prompts. Extensive experiments on USIS10K show that USIS-SAM achieves state-of-the-art performance in both class-agnostic and multi-class SIS, with ablations confirming the contributions of UA-ViT, CA, and SFPG; transferability to related underwater tasks and a land dataset (SIS10K) is also explored. The dataset and code are released to support research in underwater vision and SAM-based segmentation under challenging environmental conditions, potentially benefiting marine exploration and robotic underwater perception.

Abstract

With the breakthrough of large models, Segment Anything Model (SAM) and its extensions have been attempted to apply in diverse tasks of computer vision. Underwater salient instance segmentation is a foundational and vital step for various underwater vision tasks, which often suffer from low segmentation accuracy due to the complex underwater circumstances and the adaptive ability of models. Moreover, the lack of large-scale datasets with pixel-level salient instance annotations has impeded the development of machine learning techniques in this field. To address these issues, we construct the first large-scale underwater salient instance segmentation dataset (USIS10K), which contains 10,632 underwater images with pixel-level annotations in 7 categories from various underwater scenes. Then, we propose an Underwater Salient Instance Segmentation architecture based on Segment Anything Model (USIS-SAM) specifically for the underwater domain. We devise an Underwater Adaptive Visual Transformer (UA-ViT) encoder to incorporate underwater domain visual prompts into the segmentation network. We further design an out-of-the-box underwater Salient Feature Prompter Generator (SFPG) to automatically generate salient prompters instead of explicitly providing foreground points or boxes as prompts in SAM. Comprehensive experimental results show that our USIS-SAM method can achieve superior performance on USIS10K datasets compared to the state-of-the-art methods. Datasets and codes are released on https://github.com/LiamLian0727/USIS10K.
Paper Structure (24 sections, 6 equations, 10 figures, 9 tables)

This paper contains 24 sections, 6 equations, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Segment Anything Model (SAM)
  4. Salient Instance Segmentation (SIS)
  5. USIS10K Dataset
  6. Dataset Collection and Annotation
  7. Dataset Characteristics and Statistics
  8. USIS-SAM
  9. Underwater Adaptive ViT Encoder
  10. Salient Feature Prompt Generator (SFPG)
  11. Experiments
  12. Implementation Details
  13. Experimental Results
  14. Ablation Studies
  15. Discussion
  16. ...and 9 more sections

Figures (10)

  • Figure 1: A simple comparison of USIS-SAM and other state-of-the-art methods trained on the USIS10K dataset. Different colors represent different salient instances. URank represents the underwater image enhancement method in UnderwaterRanker underwaterranker_aaai_2023.
  • Figure 2: Examples of annotations for various salient instances in USIS10K. The image on the left is the original image and the right is the annotation mask, different colors represent different salient instances. More dataset showings can be found in \ref{['subsec:more_sample_ill']}.
  • Figure 3: Essential characteristics of the USIS10K dataset. (a) The number of salient instances per category in the USIS10K dataset. (b) Distribution of the number of salient instances per image in the USIS10K dataset. (c) Comparison of USIS10K and SIS10K in global color contrast and local color contrast.
  • Figure 4: A set of salient maps from our dataset and SIS10K.
  • Figure 5: Average channel intensity in USIS10K with proportion.
  • ...and 5 more figures