OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models

TL;DR

This work tackles open-vocabulary audio-visual segmentation by proposing OpenAVS, a training-free, language-based framework that aligns audio, visual, and textual modalities through a three-stage pipeline: audio-to-text prompt generation, LLM-guided prompt translation, and text-to-visual sounding-object segmentation. By leveraging Audio Language Models, Large Language Models, and vision foundation models (Grounded-SAM and variants), OpenAVS achieves robust open-set segmentation without labeled data and is further extended with OpenAVS-ST to exploit unlabeled data via pseudo-label self-training. The approach demonstrates state-of-the-art performance among unsupervised methods and strong generalization in few-shot and zero-shot settings across AVSBench datasets, with quantitative gains in mIoU and F-score. The findings highlight the practical impact of cross-modal, language-guided grounding for scalable, adaptable audio-visual understanding in real-world settings.

Abstract

Audio-visual segmentation aims to separate sounding objects from videos by predicting pixel-level masks based on audio signals. Existing methods primarily concentrate on closed-set scenarios and direct audio-visual alignment and fusion, which limits their capability to generalize to new, unseen situations. In this paper, we propose OpenAVS, a novel training-free language-based approach that, for the first time, effectively aligns audio and visual modalities using text as a proxy for open-vocabulary Audio-Visual Segmentation (AVS). Equipped with multimedia foundation models, OpenAVS directly infers masks through 1) audio-to-text prompt generation, 2) LLM-guided prompt translation, and 3) text-to-visual sounding object segmentation. The objective of OpenAVS is to establish a simple yet flexible architecture that relies on the most appropriate foundation models by fully leveraging their capabilities to enable more effective knowledge transfer to the downstream AVS task. Moreover, we present a model-agnostic framework OpenAVS-ST that enables the integration of OpenAVS with any advanced supervised AVS model via pseudo-label based self-training. This approach enhances performance by effectively utilizing large-scale unlabeled data when available. Comprehensive experiments on three benchmark datasets demonstrate the superior performance of OpenAVS. It surpasses existing unsupervised, zero-shot, and few-shot AVS methods by a significant margin, achieving absolute performance gains of approximately 9.4% and 10.9% in mIoU and F-score, respectively, in challenging scenarios.

Figures (7)

• Figure 1: Different ways for audio-visual alignment. (a) Embedding-based methods model audio-visual correlations directly in a latent space. (b) Language-based methods provide semantic-level alignment, enabling more effective knowledge transfer from text-audio/visual foundation models, strengthened by their superior generalization capability.
• Figure 2: System overview of OpenAVS, with prompt design for Pengi and GPT-4o presented.
• Figure 3: Segmentation improvement by translate (b) direct ALM output $\mathbf{t}_i^{(a)}$ to (c) $\hat{\mathbf{t}}_i^{(a)}$: "guitar".
• Figure 4: Illustration of the proposed prompt consistency and frame consistency in LLM-guided prompt translator.
• Figure 5: Ablation study examining the impact of box threshold variation on segmentation performance (mIoU and F-score) across S4 and MS3.