Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

Khanh-Binh Nguyen, Chae Jung Park

TL;DR

This work tackles the challenge of precise audio-visual segmentation (AVS) by enabling SAM to operate effectively in a multimodal setting. It introduces SAVE, a lightweight SAM-based model that freezes the image encoder and adds an image encoder adapter plus a residual audio encoder adapter to fuse audio cues into the transformer blocks, producing a sparse prompt for the mask decoder. With input resolution reduced to $256\times256$, SAVE achieves state-of-the-art performance on AVSBench (e.g., $M_{J}$ of $84.06$ on S4 and $64.16$ on MS3, further improved to $86.16$ and $70.83$ after synthetic pretraining and fine-tuning) and demonstrates strong zero-shot and few-shot generalization, while offering substantial gains in training and inference efficiency. The approach shows practical impact for real-world AVS tasks by reducing computational demands without sacrificing segmentation quality, enabling broader deployment across surveillance, video editing, and robotics scenarios.

Abstract

The primary aim of Audio-Visual Segmentation (AVS) is to precisely identify and locate auditory elements within visual scenes by accurately predicting segmentation masks at the pixel level. Achieving this involves comprehensively considering data and model aspects to address this task effectively. This study presents a lightweight approach, SAVE, which efficiently adapts the pre-trained segment anything model (SAM) to the AVS task. By incorporating an image encoder adapter into the transformer blocks to better capture the distinct dataset information and proposing a residual audio encoder adapter to encode the audio features as a sparse prompt, our proposed model achieves effective audio-visual fusion and interaction during the encoding stage. Our proposed method accelerates the training and inference speed by reducing the input resolution from 1024 to 256 pixels while achieving higher performance compared with the previous SOTA. Extensive experimentation validates our approach, demonstrating that our proposed model outperforms other SOTA methods significantly. Moreover, leveraging the pre-trained model on synthetic data enhances performance on real AVSBench data, achieving 84.59 mIoU on the S4 (V1S) subset and 70.28 mIoU on the MS3 (V1M) set with only 256 pixels for input images. This increases up to 86.16 mIoU on the S4 (V1S) and 70.83 mIoU on the MS3 (V1M) with inputs of 1024 pixels.

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

TL;DR

This work tackles the challenge of precise audio-visual segmentation (AVS) by enabling SAM to operate effectively in a multimodal setting. It introduces SAVE, a lightweight SAM-based model that freezes the image encoder and adds an image encoder adapter plus a residual audio encoder adapter to fuse audio cues into the transformer blocks, producing a sparse prompt for the mask decoder. With input resolution reduced to , SAVE achieves state-of-the-art performance on AVSBench (e.g., of on S4 and on MS3, further improved to and after synthetic pretraining and fine-tuning) and demonstrates strong zero-shot and few-shot generalization, while offering substantial gains in training and inference efficiency. The approach shows practical impact for real-world AVS tasks by reducing computational demands without sacrificing segmentation quality, enabling broader deployment across surveillance, video editing, and robotics scenarios.

Abstract

The primary aim of Audio-Visual Segmentation (AVS) is to precisely identify and locate auditory elements within visual scenes by accurately predicting segmentation masks at the pixel level. Achieving this involves comprehensively considering data and model aspects to address this task effectively. This study presents a lightweight approach, SAVE, which efficiently adapts the pre-trained segment anything model (SAM) to the AVS task. By incorporating an image encoder adapter into the transformer blocks to better capture the distinct dataset information and proposing a residual audio encoder adapter to encode the audio features as a sparse prompt, our proposed model achieves effective audio-visual fusion and interaction during the encoding stage. Our proposed method accelerates the training and inference speed by reducing the input resolution from 1024 to 256 pixels while achieving higher performance compared with the previous SOTA. Extensive experimentation validates our approach, demonstrating that our proposed model outperforms other SOTA methods significantly. Moreover, leveraging the pre-trained model on synthetic data enhances performance on real AVSBench data, achieving 84.59 mIoU on the S4 (V1S) subset and 70.28 mIoU on the MS3 (V1M) set with only 256 pixels for input images. This increases up to 86.16 mIoU on the S4 (V1S) and 70.83 mIoU on the MS3 (V1M) with inputs of 1024 pixels.
Paper Structure (29 sections, 4 equations, 2 figures, 9 tables)

This paper contains 29 sections, 4 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Image Encoder Adapter module
  5. Residual Audio Encoder Adapter module
  6. Fine-tuning strategy
  7. Learning Objectives
  8. Segmentation Loss.
  9. Semantic Loss.
  10. Total Loss.
  11. Experiments
  12. Datasets
  13. AVSBench dataset
  14. AVSBench-S4 subset.
  15. AVSBench-MS3 subset.
  16. ...and 14 more sections

Figures (2)

  • Figure 1: The workflow of SAVE. The image encoder is frozen while an adapter layer is placed into each transformer block to gain expertise specific to the audio-visual domain. The residual audio encoder adapter transforms the audio features and inject them into transformers block for fusing, output of the last residual audio encoder adapter is used as the sparse prompt for mask decoder.
  • Figure 2: Qualitative results produced by our model and other methods on the test sets of S4 and MS3 subsets.