Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

TL;DR

This work tackles Open-Vocabulary Semantic Segmentation (OVSS) by addressing the lack of object-level context in training-free methods. It introduces CASS, which distills object-level contextual knowledge from Vision Foundation Models (VFMs) into CLIP’s attention through spectral graph matching and low-rank refinement, and further refines text embeddings with an object presence prior to align with object-specific semantics. The approach combines Energy-based Low-rank Approximation and Dynamic Eigenscaling to produce a tailored VFM graph ${\ddot{A}}_{\text{VFM}}^i$ that complements CLIP’s attention, defined via ${A}_{\psi}^j$, and uses an Object-Presence Prior to adjust patch-text similarities for cohesive object-level masks. Across multiple datasets (e.g., VOC, Context, COCO-Stuff, ADE20K, Cityscapes), CASS achieves state-of-the-art performance among training-free OVSS methods, demonstrating robust generalization to unseen classes and real-world scenes, with practical implications for tasks like image inpainting and object removal.

Abstract

Open-Vocabulary Semantic Segmentation (OVSS) has advanced with recent vision-language models (VLMs), enabling segmentation beyond predefined categories through various learning schemes. Notably, training-free methods offer scalable, easily deployable solutions for handling unseen data, a key goal of OVSS. Yet, a critical issue persists: lack of object-level context consideration when segmenting complex objects in the challenging environment of OVSS based on arbitrary query prompts. This oversight limits models' ability to group semantically consistent elements within object and map them precisely to user-defined arbitrary classes. In this work, we introduce a novel approach that overcomes this limitation by incorporating object-level contextual knowledge within images. Specifically, our model enhances intra-object consistency by distilling spectral-driven features from vision foundation models into the attention mechanism of the visual encoder, enabling semantically coherent components to form a single object mask. Additionally, we refine the text embeddings with zero-shot object presence likelihood to ensure accurate alignment with the specific objects represented in the images. By leveraging object-level contextual knowledge, our proposed approach achieves state-of-the-art performance with strong generalizability across diverse datasets.

Figures (15)

• Figure 1: We present CASS, object-level Context-Aware training-free open-vocabulary Semantic Segmentation model. (a) Overview: Our method distills the vision foundation model's (VFM) object-level contextual spectral graph into CLIP’s attention and refines query text embeddings towards object-specific semantics. (b) Object-Level Context: This result illustrates how incorporating object-level context improves segmentation accuracy by unifying object-wise components into a user-defined object class; for example, (top row) our approach precisely segments the truck’s body, wheels, and cargo area and (bottom row) accurately groups elements into a single object, such as the sheep, whereas baselines hajimiri2025nacliplan2024proxyclip often fail to achieve this unified segmentation.
• Figure 2: Detailed illustration of our proposed training-free spectral object-level context distillation mechanism in Sec. \ref{['sec:method_local']}. By matching the attention graphs of VFM and CLIP head-by-head to establish complementary relationships, and distilling the fundamental object-level context of the VFM graph to CLIP, we enhance CLIP’s ability to capture intra-object contextual coherence.
• Figure 3: Attention score visualization for various query points. Left: Vanilla CLIP (${A}_{\text{CLIP}}$) shows noisy, unfocused attention. Center: VFM-to-CLIP distillation without low-rank eigenscaling shows partial object grouping with limited detail. Right: Incorporating our low-rank eigenscaling (Eq. \ref{['eq:distillation']}) captures object-level context, improving grouping within a single object.
• Figure 4: Visualization of the matched complementary graph pairs. The red dotted line indicates the focus areas of VFM, while the yellow dotted line highlights those for CLIP. By aggregating graphs with contrasting structural properties, we enhance the object-level context in ${A}_\text{CLIP}$ supported by $\ddot{ {A}}_\text{VFM}$, resulting ${A}_\psi$.
• Figure 5: Detailed illustration of our object presence prior-guided text embedding adjustment module. The CLIP text encoder generates text embeddings for each object class, and the object presence prior is derived from both visual and text embeddings. Within hierarchically defined class groups, text embeddings are selected based on object presence prior, then refined in an object-specific direction to align with components likely present in the image.