Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation

TL;DR

The paper addresses the challenge of training-free open-vocabulary semantic segmentation (OVSS) using CLIP, which is biased toward global image-text alignment and lacks fine-grained local discrimination. It introduces a structure-aware feature rectification framework that builds a Region Adjacency Graph (RAG) from low-level color and texture cues to guide patch-level attention (RAG-guided Attention) and to refine cross-modal similarities (Similarity Fusion). By encoding local structure with superpixel-based RAGs and integrating bilateral attention biases, the method reduces segmentation noise and improves region-level consistency across multiple OVSS benchmarks without additional training. Extensive ablations and experiments demonstrate gains across datasets, robustness to perturbations, and insights into hyperparameters and RAG design, establishing a practical, generalizable approach to open-vocabulary segmentation. The work highlights the value of leveraging low-level priors to constrain high-level vision-language alignment in dense prediction tasks.

Abstract

Benefiting from the inductive biases learned from large-scale datasets, open-vocabulary semantic segmentation (OVSS) leverages the power of vision-language models, such as CLIP, to achieve remarkable progress without requiring task-specific training. However, due to CLIP's pre-training nature on image-text pairs, it tends to focus on global semantic alignment, resulting in suboptimal performance when associating fine-grained visual regions with text. This leads to noisy and inconsistent predictions, particularly in local areas. We attribute this to a dispersed bias stemming from its contrastive training paradigm, which is difficult to alleviate using CLIP features alone. To address this, we propose a structure-aware feature rectification approach that incorporates instance-specific priors derived directly from the image. Specifically, we construct a region adjacency graph (RAG) based on low-level features (e.g., colour and texture) to capture local structural relationships and use it to refine CLIP features by enhancing local discrimination. Extensive experiments show that our method effectively suppresses segmentation noise, improves region-level consistency, and achieves strong performance on multiple open-vocabulary segmentation benchmarks.

Figures (15)

• Figure 1: Illustration of the main idea and performance. High-level feature region adjacency graphs (RAGs) introduce local noise, while low-level colour-based RAGs maintain clean structure. The RAGs built on CLIP clip and DINO dino pretrained features exhibit noisy and inconsistent connectivity in local regions (see zoomed-in areas), when compared to the low-level based one. This highlights the potential of low-level cues for tasks requiring fine-grained local modelling, e.g. image segmentation. Bottom: Comparison of average performance across multiple datasets using different features for RAG construction. C.-only: colour-based features, and C. + G.: colour and texture features.
• Figure 2: Illustration of superpixel-to-patch encoding. The distance between two patches is first represented as a list of all pairwise superpixel regions $(\textcolor{red}{\boldsymbol{\Box}},\textcolor{blue}{\boldsymbol{\Box}})$, then the patch distance is computed from this list using Eq. \ref{['dis_patch']}.
• Figure 3: Illustration of different attention bias mechanisms. The first column shows the input images. The second column visualises the traditional Gaussian kernel, which models spatial proximity in a local window. The third column shows the RAG-bias computed from the Region Adjacency Graph (RAG), capturing structural relationships between neighbouring regions. The fourth column combines both the Gaussian kernel and the RAG-bias to form a bilateral attention bias, which accounts for both spatial distance and local structure.
• Figure 4: Overview of the proposed RAG-guided attention mechanism. The bilateral attention bias is computed by combining a spatial Gaussian kernel with a structure-aware RAG-bias. This combined bias is integrated into the attention weights to enhance structural sensitivity. Right: visualisation of the Gaussian and bilateral attention maps.
• Figure 5: The qualitative results of our method. For more challenging cases, such as grayscale and stylised images (e.g. oil paintings), please refer to Figs. \ref{['fig:lighting']}, \ref{['fig:destruction']}, and \ref{['fig:domain_shift']} in the Supplementary Materials.