Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAS-SAM: Segment Any Marine Animal with Aggregated Features

Tianyu Yan, Zifu Wan, Xinhao Deng, Pingping Zhang, Yang Liu, Huchuan Lu

TL;DR

This paper tackles marine animal segmentation in challenging underwater environments by adapting the Segment Anything Model (SAM) through an Adapter-informed SAM Encoder (ASE), a Hypermap Extraction Module (HEM), and a Progressive Prediction Decoder (PPD) with a Fusion Attention Module (FAM). The approach injects marine-domain knowledge via LoRA in MHSA and an Adapter in FFN, extracts multi-scale guidance with HEM, and progressively fuses prompts, ASE, and HEM in the PPD to recover fine-grained masks. Extensive experiments on four MAS datasets show state-of-the-art performance and robust generalization, with ablations validating the contribution of each module and the benefits of deep supervision. The proposed MAS-SAM has practical implications for underwater robotics and marine biology by enabling more accurate, scalable segmentation of camouflage and variable-shape marine life from complex underwater imagery.

Abstract

Recently, Segment Anything Model (SAM) shows exceptional performance in generating high-quality object masks and achieving zero-shot image segmentation. However, as a versatile vision model, SAM is primarily trained with large-scale natural light images. In underwater scenes, it exhibits substantial performance degradation due to the light scattering and absorption. Meanwhile, the simplicity of the SAM's decoder might lead to the loss of fine-grained object details. To address the above issues, we propose a novel feature learning framework named MAS-SAM for marine animal segmentation, which involves integrating effective adapters into the SAM's encoder and constructing a pyramidal decoder. More specifically, we first build a new SAM's encoder with effective adapters for underwater scenes. Then, we introduce a Hypermap Extraction Module (HEM) to generate multi-scale features for a comprehensive guidance. Finally, we propose a Progressive Prediction Decoder (PPD) to aggregate the multi-scale features and predict the final segmentation results. When grafting with the Fusion Attention Module (FAM), our method enables to extract richer marine information from global contextual cues to fine-grained local details. Extensive experiments on four public MAS datasets demonstrate that our MAS-SAM can obtain better results than other typical segmentation methods. The source code is available at https://github.com/Drchip61/MAS-SAM.

MAS-SAM: Segment Any Marine Animal with Aggregated Features

TL;DR

This paper tackles marine animal segmentation in challenging underwater environments by adapting the Segment Anything Model (SAM) through an Adapter-informed SAM Encoder (ASE), a Hypermap Extraction Module (HEM), and a Progressive Prediction Decoder (PPD) with a Fusion Attention Module (FAM). The approach injects marine-domain knowledge via LoRA in MHSA and an Adapter in FFN, extracts multi-scale guidance with HEM, and progressively fuses prompts, ASE, and HEM in the PPD to recover fine-grained masks. Extensive experiments on four MAS datasets show state-of-the-art performance and robust generalization, with ablations validating the contribution of each module and the benefits of deep supervision. The proposed MAS-SAM has practical implications for underwater robotics and marine biology by enabling more accurate, scalable segmentation of camouflage and variable-shape marine life from complex underwater imagery.

Abstract

Recently, Segment Anything Model (SAM) shows exceptional performance in generating high-quality object masks and achieving zero-shot image segmentation. However, as a versatile vision model, SAM is primarily trained with large-scale natural light images. In underwater scenes, it exhibits substantial performance degradation due to the light scattering and absorption. Meanwhile, the simplicity of the SAM's decoder might lead to the loss of fine-grained object details. To address the above issues, we propose a novel feature learning framework named MAS-SAM for marine animal segmentation, which involves integrating effective adapters into the SAM's encoder and constructing a pyramidal decoder. More specifically, we first build a new SAM's encoder with effective adapters for underwater scenes. Then, we introduce a Hypermap Extraction Module (HEM) to generate multi-scale features for a comprehensive guidance. Finally, we propose a Progressive Prediction Decoder (PPD) to aggregate the multi-scale features and predict the final segmentation results. When grafting with the Fusion Attention Module (FAM), our method enables to extract richer marine information from global contextual cues to fine-grained local details. Extensive experiments on four public MAS datasets demonstrate that our MAS-SAM can obtain better results than other typical segmentation methods. The source code is available at https://github.com/Drchip61/MAS-SAM.
Paper Structure (16 sections, 19 equations, 5 figures, 3 tables)

This paper contains 16 sections, 19 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Marine Animal Segmentation
  4. SAM for Customized Tasks
  5. Proposed Method
  6. Adapter-informed SAM Encoder
  7. Hypermap Extraction Module
  8. Progressive Prediction Decoder
  9. Model Training
  10. Experiments
  11. Datasets and Evaluation Metrics
  12. Implementation Details
  13. Comparison with State-of-the-arts
  14. Ablation Studies
  15. More Discussions
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The overall structure of our proposed framework (MAS-SAM). It consists of three main components: Adapter-informed SAM Encoder (ASE), Hypermap Extraction Module (HEM) and Progressive Prediction Decoder (PPD).
  • Figure 2: The enhanced Transformer block in our proposed Adapter-informed SAM Encoder.
  • Figure 3: The structure of our Fusion Attention Module.
  • Figure 4: Visual comparison of predicted segmentation masks with different methods. The images in the 1-2nd rows are from the MAS3K, 3-4th are from RMAS, 5-6th are from UFO120 and the 7-8th are from RUWI. Best view by zooming in.
  • Figure 5: Visual comparison of predicted segmentation masks with different modules. Best view by zooming in.