Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation

TL;DR

This work proposes a strong baseline for training-free OVSS, termed Neighbour-Aware CLIP (NACLIP), representing a straightforward adaptation of CLIP tailored for this scenario, and enforces localization of patches in the self-attention of CLIP's vision transformer which, despite being crucial for dense prediction tasks, has been overlooked in the OVSS literature.

Abstract

Despite the significant progress in deep learning for dense visual recognition problems, such as semantic segmentation, traditional methods are constrained by fixed class sets. Meanwhile, vision-language foundation models, such as CLIP, have showcased remarkable effectiveness in numerous zero-shot image-level tasks, owing to their robust generalizability. Recently, a body of work has investigated utilizing these models in open-vocabulary semantic segmentation (OVSS). However, existing approaches often rely on impractical supervised pre-training or access to additional pre-trained networks. In this work, we propose a strong baseline for training-free OVSS, termed Neighbour-Aware CLIP (NACLIP), representing a straightforward adaptation of CLIP tailored for this scenario. Our method enforces localization of patches in the self-attention of CLIP's vision transformer which, despite being crucial for dense prediction tasks, has been overlooked in the OVSS literature. By incorporating design choices favouring segmentation, our approach significantly improves performance without requiring additional data, auxiliary pre-trained networks, or extensive hyperparameter tuning, making it highly practical for real-world applications. Experiments are performed on 8 popular semantic segmentation benchmarks, yielding state-of-the-art performance on most scenarios. Our code is publicly available at https://github.com/sinahmr/NACLIP.

Figures (7)

• Figure 1: Attention maps of the final visual encoder layer. For the patches shaded in red (denoted with (a) to (d)), the final layer's attention maps are presented for CLIP clip, SCLIP sclip, and our method. We have identified two problematic phenomena in the attention maps of CLIP and SCLIP, stemming from a lack of mechanisms to properly attend to patches' neighbourhoods. First, as depicted in (a) and (b), attention intensity is sometimes dispersed among distant patches, neglecting the vicinity of a patch. Additionally, adjacent or closely located patches sharing the same real-world category and even similar visual characteristics can have inconsistent attention maps. For instance, while SCLIP generates a quality attention map for patch (d), its attention map for (c) is notably different and fails to focus on the desired object. By explicitly promoting attention to neighbours, our method produces consistent attention maps across adjacent patches.
• Figure 2: Schematic figure depicting the mechanism to form attention maps. Maps are shown for the patch located at $\color{red} \bigstar$, $\omega({\color{red} \bigstar})$ denotes a discretized Gaussian kernel centered at $\color{red} \bigstar$ (see \ref{['eq:gaussian_window']}), and $[{\mathbf{x} \mathbf{y}^{\top}}]_{\color{red} \bigstar}$ indexes $\mathbf{x} \mathbf{y}^{\top}$ on patch $\color{red} \bigstar$. CLIP does not ensure high attention to the patch itself and the neighbouring patches, while NACLIP does. Scaling and $\mathrm{softmax}$ operations are omitted for demonstration simplicity.
• Figure 3: Qualitative results (segmentation maps) on PASCAL Context (59) pascalcontext for CLIP clip, SCLIP sclip, and our method.
• Figure 4: Illustrative example of \ref{['app:eq:std-circle']}. The attention logits of the center point to the points within the depicted circle are increased by at least $\tau$. Example generated for $\sigma = 5$ and $\tau = 0.8$.
• Figure 5: Ablation study on the impact of $\sigma$. We have provided results for both cases of using and not using post-processing, revealing consistent trends across both cases.