AURORA:Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation

TL;DR

AURORA tackles Reference Audio-Visual Segmentation by enforcing genuine multimodal reasoning through structured Chain-of-Thought prompts, while preserving pixel-precise segmentation via a segmentation feature distillation loss. It then advances reasoning quality with a corrective reflective-style training stage and reinforces robustness with Group Reward Policy Optimization, incorporating format, IoU, and class rewards. The approach sets new state-of-the-art on Ref-AVS and demonstrates strong cross-task generalization to unreferenced AVS, highlighting substantial improvements in both semantic grounding and segmentation accuracy. This framework underscores the viability of integrating deliberate reasoning processes into vision-language models for fine-grained multimodal tasks.

Abstract

Reference Audio-Visual Segmentation (Ref-AVS) tasks challenge models to precisely locate sounding objects by integrating visual, auditory, and textual cues. Existing methods often lack genuine semantic understanding, tending to memorize fixed reasoning patterns. Furthermore, jointly training for reasoning and segmentation can compromise pixel-level precision. To address these issues, we introduce AURORA, a novel framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation. We employ a structured Chain-of-Thought (CoT) prompting mechanism to guide the model through a step-by-step reasoning process and introduce a novel segmentation feature distillation loss to effectively integrate these reasoning abilities without sacrificing segmentation performance. To further cultivate the model's genuine reasoning capabilities, we devise a further two-stage training strategy: first, a ``corrective reflective-style training" stage utilizes self-correction to enhance the quality of reasoning paths, followed by reinforcement learning via Group Reward Policy Optimization (GRPO) to bolster robustness in challenging scenarios. Experiments demonstrate that AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.

Figures (12)

• Figure 1: Overall training pipeline of our proposed model. The training pipeline consists of three stages: (1) Supervised Fine-Tuning with CoT paths ($\bm{y}_t$) generated by Qwen-Omni. (2) Corrective Reflective-Style Training, in which we construct a "reflective-style" path by combining the reasoning output from the SFT-trained model, a self-correction trigger, and the corrected path ($\bm{y}_{correct}$) from Gemini. (3) Reinforcement Learning via GRPO to further refine the model's reasoning.
• Figure 2: Overall framework of our proposed model. Our model integrates SAM and VideoLLaMA2. The gray block represents the Segmentation Feature Distillation Loss during the SFT stage, and the green block denotes the GRPO process with triplet rewards in Stage 3.
• Figure 3: The visualization results of the referred objects in the Ref-AVS compared with EEMC wang2024ref. Note that although the reasoning steps may appear in different orders to enhance diversity and support GRPO exploration, all outputs consistently contain the four key reasoning steps shown in Figure \ref{['prompt_FIG']}.
• Figure 4: A Taxonomy of Analyzed Failure Cases from the Baseline Qwen-Omni Model. It illustrates the categorical distribution of errors within our curated analysis set. The significant proportion of Knowledge Misapplication and Perceptual Failure provides empirical evidence that standard SFT with CoT training is insufficient to address these deep-seated and foundational error types.
• Figure 5: Example of Knowledge Misapplication, which consists of Over-reliance on General Knowledge and Visual Object Misidentification.