Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation

TL;DR

This paper tackles training-free video reasoning segmentation by leveraging Multimodal Large Language Models (MLLMs) and their attention mechanisms. It casts video QA grounding as a grounding problem and refines attention rollout maps through Decomposed Attention Fusion (DecAF), which combines Contrastive Object-Background Fusion and Complementary Video-Frame Fusion to produce robust coarse masks. To obtain fine-grained segmentation without retraining, it adds an attention-guided SAM2 prompting pipeline that uses point queries derived from attention maps, propagates masks across frames, and scores tracklets with $s_{i}^{trk} = \text{Avg}(\mathbf{V}_{p_i}, s_{i}^{\text{SAM}}, s^{ac})$, where $s^{ac}$ measures attention consistency. Across RVOS and ReasonVOS benchmarks, DecAF consistently outperforms training-free baselines and approaches training-based methods, highlighting the practical viability of training-free localization for complex video reasoning tasks, with no model retraining required.

Abstract

Multimodal large language models (MLLMs) demonstrate strong video understanding by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-free manner, we cast video reasoning segmentation as a video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive object-background fusion and (2) complementary video-frame fusion. This method suppresses irrelevant activations and enhances object-focused cues, enabling direct conversion of attention maps into coarse segmentation masks. In addition, we introduce attention-guided SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MLLMs with SAM, our method operates entirely without retraining. DecAF outperforms training-free methods and achieves performance comparable to training-based methods on both referring and reasoning VOS benchmarks. The code will be available at https://github.com/HYUNJS/DecAF.

Figures (10)

• Figure 1: Visualization of our method. (a) Noise in irrelevant regions is suppressed by contrastive fusion with the background attention map. As shown in the first frame, background activations are removed, and the target object is emphasized. (b) Video attention map captures temporal cues, while frame attention map highlights object-centric details. Their fusion resolves conflicts (e.g., identifying the server vs. the hitting player) and produces more consistent localization. The attention mask is obtained directly from the attention map while the SAM mask is generated by SAM2.
• Figure 2: Overview of DecAF. (a) Attention rollout with our V-Max normalization produces a rollout matrix that accumulates attention across layers, from which visual-token scores for the final query token are extracted as attention maps for grounding. (b) Contrastive fusion suppresses attention scores on background regions. (c) Complementary fusion integrates video- and frame-level cues. (d) These fusion methods are combined into the full pipeline to refine noisy attention maps.
• Figure 3: Overview of our SAM prompting pipeline with attention maps. (1) Point queries for SAM2 are obtained from attention maps via thresholding ($\tau_{pq}$). (2) During mask propagation, highly overlapping masks are removed. (3) Spurious mask tracklets are removed using our scoring method.
• Figure 4: Analysis of TAM's failure cases
• Figure 5: Qualitative results for the single object case.