Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective

TL;DR

This work identifies a distraction phenomenon in CLIP’s dense prediction for open-vocabulary semantic segmentation, where certain tokens with high dimension-weight activations siphon attention away from target regions. It introduces ReFocusing CLIP (RF-CLIP), a training-free method that localizes distraction tokens and defocused targets, then redistributes attention and embedding resources to restore precise pixel-level multimodal alignment. Through distractor localization, defocus localization via graph-cut, and topology-aware weight redistribution, RF-CLIP delivers state-of-the-art mIoU across eight OVSS benchmarks while preserving high inference efficiency. The approach highlights the importance of intermediate-layer alignment and presents a practical, model-agnostic strategy to improve dense predictions in vision-language models.

Abstract

Open-vocabulary semantic segmentation (OVSS) employs pixel-level vision-language alignment to associate category-related prompts with corresponding pixels. A key challenge is enhancing the multimodal dense prediction capability, specifically this pixel-level multimodal alignment. Although existing methods achieve promising results by leveraging CLIP's vision-language alignment, they rarely investigate the performance boundaries of CLIP for dense prediction from an interpretability mechanisms perspective. In this work, we systematically investigate CLIP's internal mechanisms and identify a critical phenomenon: analogous to human distraction, CLIP diverts significant attention resources from target regions to irrelevant tokens. Our analysis reveals that these tokens arise from dimension-specific over-activation; filtering them enhances CLIP's dense prediction performance. Consequently, we propose ReFocusing CLIP (RF-CLIP), a training-free approach that emulates human distraction-refocusing behavior to redirect attention from distraction tokens back to target regions, thereby refining CLIP's multimodal alignment granularity. Our method achieves SOTA performance on eight benchmarks while maintaining high inference efficiency.

Figures (15)

• Figure 1: RF-CLIP achieves precise attention focus, which facilitates accurate segmentation of target regions. Compared to state-of-the-art methods, RF-CLIP demonstrates superior performance, achieving the highest accuracy.
• Figure 2: Illustration of "distraction" phenomenon. (a) Layer-wise attention maps for two query locations reveal query-to-tokens relevance across the entire image. (b) Layer-wise head-averaged self-attention maps characterize token-to-token relationships. Red solid boxes denote query points; white dashed boxes indicate distraction tokens; and red dashed boxes represent attention weights of distraction tokens.
• Figure 3: Weights analysis. (a) Histogram of per-dimension mean embedding weights averaged across the entire dataset. (b) Scatter plot of tokens' attention weights versus their maximum embedding weights in distraction dimensions.
• Figure 4: Distraction tokens identification. (a) Scatter plot of attention weights versus maximum embedding weights in $\mathcal{D}_{dis}$. (b) Layer-wise attention maps and above-threshold tokens visualizations.
• Figure 5: Overview of RF-CLIP. Our RF-CLIP directly correct layer-wise spatial misalignment in CLIP to enhance dense prediction capabilities. Each layer's correction mechanism comprises three key components: (a) distractor localization, identifying attention-rich distraction tokens irrelevant to target objects; (b) defocus localization, detecting attention-poor defocused tokens relevant to targets; (c) weight redistribution, reallocating attention resources from distraction tokens to defocused tokens.