NERVE: Neighbourhood & Entropy-guided Random-walk for training free open-Vocabulary sEgmentation

TL;DR

NERVE tackles open-vocabulary semantic segmentation in a training-free setting by fusing global diffusion-attention with local spatial cues through a stochastic random-walk refinement. It models segmentation as a probabilistic diffusion on image tokens, using $S = \beta S_{global} + (1-\beta) S_{local}$ and a node-to-label matrix $G$ derived from CLIP cross-attention, yielding $P_{\infty} = (1-\alpha)(I - \alpha S)^{-1}G$. To handle multi-head attention robustly, it introduces entropy-based weighting across heads, forming $A_{weighted} = \sum_h w_h A^{(h)}$ with $w_h$ derived from $\mathcal{H}^{(h)}$, thereby suppressing noisy maps. Efficient computation relies on a truncated random-walk update $\tilde{P}_L = (1-\alpha)G + \alpha S\tilde{P}_{L-1}$ and normalization, exploiting the low-rank/global and sparse local structures to achieve favorable complexity. Across five OVSS benchmarks, NERVE achieves strong zero-shot performance without post-processing, demonstrating the effectiveness of entropy-guided, neighborhood-aware diffusion for open-vocabulary segmentation.

Abstract

Despite recent advances in Open-Vocabulary Semantic Segmentation (OVSS), existing training-free methods face several limitations: use of computationally expensive affinity refinement strategies, ineffective fusion of transformer attention maps due to equal weighting or reliance on fixed-size Gaussian kernels to reinforce local spatial smoothness, enforcing isotropic neighborhoods. We propose a strong baseline for training-free OVSS termed as NERVE (Neighbourhood \& Entropy-guided Random-walk for open-Vocabulary sEgmentation), which uniquely integrates global and fine-grained local information, exploiting the neighbourhood structure from the self-attention layer of a stable diffusion model. We also introduce a stochastic random walk for refining the affinity rather than relying on fixed-size Gaussian kernels for local context. This spatial diffusion process encourages propagation across connected and semantically related areas, enabling it to effectively delineate objects with arbitrary shapes. Whereas most existing approaches treat self-attention maps from different transformer heads or layers equally, our method uses entropy-based uncertainty to select the most relevant maps. Notably, our method does not require any conventional post-processing techniques like Conditional Random Fields (CRF) or Pixel-Adaptive Mask Refinement (PAMR). Experiments are performed on 7 popular semantic segmentation benchmarks, yielding an overall state-of-the-art zero-shot segmentation performance, providing an effective approach to open-vocabulary semantic segmentation.

Figures (3)

• Figure 1: Overview of our training-free open-vocabulary segmentation pipeline. Given an image and a set of text prompts, we extract cross-attention maps using CLIP and self-attention maps from a Stable Diffusion encoder. We compute entropy-guided fusion across attention heads $h$ to obtain global ($A_{\text{global}}$) and local ($A_{\text{local}}$) affinities. These are normalized and linearly combined into a final stochastic matrix $S$, which is used in a truncated stochastic random walk to propagate semantic information and generate refined segmentation masks.
• Figure 2: Impact of random‑walk iteration count on segmentation mIoU. Evolution of mIoU on PASCAL VOC, VOC21, and PC59 as a function of the number of stochastic random‑walk refinement steps, demonstrating rapid gains up to 20 iterations and saturation thereafter.
• Figure 3: Progressive segmentation refinement via Stochastic Random Walk. For each image, we visualize the effect of increasing random walk steps on the segmentation map. Regions of interest are highlighted with colored boxes. Our method produces increasingly accurate and coherent segmentations over time, illustrating strong local–global propagation.

Theorems & Definitions (4)

• Theorem 3.1
• proof
• Theorem 3.1
• proof