ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentation

TL;DR

This work tackles the limitation of CLIP-based open-vocabulary semantic segmentation by addressing CLIP’s coarse pixel-level localization. It introduces the Attention Refinement Module (ARM), a lightweight train-once post-processor that adaptively fuses CLIP’s multi-level features through semantically guided cross-attention and subsequent self-attention, producing a refined, class-agnostic map that is then projected into a class-specific refinement via text embeddings and added to the coarse CLIP output. Trained on COCO-Stuff, ARM serves as a universal plug-and-play component for training-free OVSS frameworks, delivering consistent improvements across multiple benchmarks with negligible inference overhead. Through extensive ablations, the authors show that a simple, well-structured refinement of internal CLIP cues outperforms static fusion and offers strong zero-shot transfer, highlighting the untapped potential of CLIP’s internal representations for dense prediction tasks.

Abstract

Open-vocabulary semantic segmentation (OVSS) is fundamentally hampered by the coarse, image-level representations of CLIP, which lack precise pixel-level details. Existing training-free methods attempt to resolve this by either importing priors from costly external foundation models (e.g., SAM, DINO) or by applying static, hand-crafted heuristics to CLIP's internal features. These approaches are either computationally expensive or sub-optimal. We propose the Attention Refinement Module (ARM), a lightweight, learnable module that effectively unlocks and refines CLIP's internal potential. Unlike static-fusion methods, ARM learns to adaptively fuse hierarchical features. It employs a semantically-guided cross-attention block, using robust deep features (K, V) to select and refine detail-rich shallow features (Q), followed by a self-attention block. The key innovation lies in a ``train once, use anywhere" paradigm. Trained once on a general-purpose dataset (e.g., COCO-Stuff), ARM acts as a universal plug-and-play post-processor for diverse training-free frameworks. Extensive experiments show that ARM consistently boosts baseline performance on multiple benchmarks with negligible inference overhead, establishing an efficient and effective paradigm for training-free OVSS.

Figures (4)

• Figure 1: Comparison of different approaches for training-free OVSS. (a) Methods relying on external Vision Foundation Models (VFMs) like SAM or DINO. (b) ResCLIP statically fuses intermediate attention maps. (c) Our ARM-CLIP learns to refine CLIP's hierarchical features using a lightweight trainable module.
• Figure 2: Our Attention Refinement Module (ARM) architecture. ARM extracts hierarchical features from the image encoder and fuses them sequentially through cross-attention and self-attention mechanisms. The fused features are processed by Scale blocks to generate a class-agnostic affinity map. This map is then multiplied by the text embedding ($\otimes$) to generate a class-related refined map. Finally, this refined map is added ($\oplus$) to the original coarse affinity map to correct localization errors and restore details.
• Figure 3: Zero-shot transfer results of applying ARM to CLIPer on the Context-59. The ARM trained with standard CLIP features (blue) resulted in a performance degradation, while the ARM trained with Clear-CLIP features (red) resulted in a significant performance improvement.
• Figure 4: Qualitative Results. Visual comparison on sample images from three different datasets. Compared to these methods, our method has more accurate segmentation results that are closer to the ground-truths.