Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation

Jinbae Seo, Hyeongjun Kwon, Kwonyoung Kim, Jiyoung Lee, Kwanghoon Sohn

TL;DR

The paper tackles audiovisual instance segmentation (AVIS) and the problem of visual bias that causes queries to attend to visually salient objects rather than sound sources. It introduces ACVIS, featuring an Audio-Centric Query Generator (ACQG) that uses cross-attention to produce sound-specific queries and a Sound-Aware Ordinal Counting (SAOC) loss to supervise the number of sounding objects via ordinal regression. On the AVISeg benchmark, ACVIS achieves improvements of +1.64 mAP, +0.60 HOTA, and +2.06 FSLA, validating that query specialization and counting supervision enhance audiovisual segmentation. By combining audio-driven query generation with monotonic counting constraints, the approach improves robustness in multi-source scenes and has potential applicability to broader multimodal segmentation tasks.

Abstract

Audiovisual instance segmentation (AVIS) requires accurately localizing and tracking sounding objects throughout video sequences. Existing methods suffer from visual bias stemming from two fundamental issues: uniform additive fusion prevents queries from specializing to different sound sources, while visual-only training objectives allow queries to converge to arbitrary salient objects. We propose Audio-Centric Query Generation using cross-attention, enabling each query to selectively attend to distinct sound sources and carry sound-specific priors into visual decoding. Additionally, we introduce Sound-Aware Ordinal Counting (SAOC) loss that explicitly supervises sounding object numbers through ordinal regression with monotonic consistency constraints, preventing visual-only convergence during training. Experiments on AVISeg benchmark demonstrate consistent improvements: +1.64 mAP, +0.6 HOTA, and +2.06 FSLA, validating that query specialization and explicit counting supervision are crucial for accurate audiovisual instance segmentation.

Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation

TL;DR

The paper tackles audiovisual instance segmentation (AVIS) and the problem of visual bias that causes queries to attend to visually salient objects rather than sound sources. It introduces ACVIS, featuring an Audio-Centric Query Generator (ACQG) that uses cross-attention to produce sound-specific queries and a Sound-Aware Ordinal Counting (SAOC) loss to supervise the number of sounding objects via ordinal regression. On the AVISeg benchmark, ACVIS achieves improvements of +1.64 mAP, +0.60 HOTA, and +2.06 FSLA, validating that query specialization and counting supervision enhance audiovisual segmentation. By combining audio-driven query generation with monotonic counting constraints, the approach improves robustness in multi-source scenes and has potential applicability to broader multimodal segmentation tasks.

Abstract

Audiovisual instance segmentation (AVIS) requires accurately localizing and tracking sounding objects throughout video sequences. Existing methods suffer from visual bias stemming from two fundamental issues: uniform additive fusion prevents queries from specializing to different sound sources, while visual-only training objectives allow queries to converge to arbitrary salient objects. We propose Audio-Centric Query Generation using cross-attention, enabling each query to selectively attend to distinct sound sources and carry sound-specific priors into visual decoding. Additionally, we introduce Sound-Aware Ordinal Counting (SAOC) loss that explicitly supervises sounding object numbers through ordinal regression with monotonic consistency constraints, preventing visual-only convergence during training. Experiments on AVISeg benchmark demonstrate consistent improvements: +1.64 mAP, +0.6 HOTA, and +2.06 FSLA, validating that query specialization and explicit counting supervision are crucial for accurate audiovisual instance segmentation.

Paper Structure

This paper contains 11 sections, 7 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation and Overview
  4. Audio-Centric Frame Queries
  5. Sound-Aware Ordinal Counting loss
  6. Training and Inference
  7. Experiments
  8. Experimental Settings
  9. Main Results
  10. Ablation Studies
  11. Conclusion

Figures (3)

  • Figure 1: Visual bias in audiovisual instance segmentation. While ground truth (b) indicates only one person speaking, previous work (AVISM) guo2025avis (c) detects two visible people due to visual dominance. Our ACVIS (d) correctly identifies the speaking person by maintaining audio-visual balance through specialized queries and counting supervision.
  • Figure 2: (a) Overall architecture with audio-centric object localizer and object tracker. Frame queries and count token are processed through our query generator and decoder for AVIS. (b) Our SAOC loss prevents visual bias: without our loss (left), the model over-detects visually salient objects; with our loss (right), only sounding objects are segmented.
  • Figure 3: Qualitative results across diverse audio scenarios with varying sound sources.