DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception

TL;DR

This work analyzes why CLIP struggles in open-vocabulary dense perception, identifying a proxy-token phenomenon that distributes attention away from local regions. It introduces DeCLIP, which decouples CLIP's final attention into content and context streams and distills them from image crops (content) and Vision Foundation Models (context) to improve local discriminability and spatial consistency. Across extensive open-vocabulary detection and segmentation benchmarks, DeCLIP consistently outperforms state-of-the-art methods, with ablations showing the benefits of decoupled distillation and VFM guidance. The approach enables more accurate region-level recognition and pixel-precise segmentation in open-ended visual concept spaces, advancing practical open-vocabulary vision systems.

Abstract

Dense visual prediction tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense prediction often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively. The ``content'' features are aligned with image crop representations to improve local discriminability, while ``context'' features learn to retain the spatial correlations under the guidance of vision foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP significantly outperforms existing methods across multiple open-vocabulary dense prediction tasks, including object detection and semantic segmentation. Code is available at \textcolor{magenta}{https://github.com/xiaomoguhz/DeCLIP}.

Figures (11)

• Figure 1: DeCLIP outperforms previous state-of-the-art models on a broad range of open-vocabulary dense prediction benchmarks.
• Figure 2: Quantitative and qualitative comparisons between our method and CLIP.(a) Performance comparisons of open-vocabulary dense predictions on COCO mscoco. (b) Attention map comparisons, with the anchor image token marked in red.
• Figure 3: Visualization of attention maps across different encoding layers of CLIP and VFM. The attention weights are calculated at a low resolution, then averaged across different heads, and finally upsampled to the original image resolution for visualization. The anchor image token is marked in red. We observe the occurrence of the "proxy" token phenomenon in CLIP, but not in VFM. Furthermore, when the position of the anchor image token is shifted, VFM shows a better correlation for image tokens with the same semantics.
• Figure 4: Pre-fine-tuning methods for adapting CLIP to dense prediction tasks. Existing work considers establishing region-text alignment through cost-effective methods via: (a) using images as pseudo regions or (b) using self-distillation on image patches. The former regards the entire image as a region, which results in a loss of details. The latter uses self-distillation on the image patches thereby gaining more fine-grained information, but still fails to apply to pixel-level image segmentation. (c) Unlike prior approaches, we use VFM to guide the spatial consistency of CLIP's features, and decouple CLIP's features for distillation separately to avoid optimization conflicts.
• Figure 5: Illustration of the DeCLIP framework. We decouple CLIP's final attention module into context and content features for distillation, avoiding optimization conflicts between feature correlations and visual-language alignment. CLIP itself serves as the teacher for content features to improve region classification accuracy. A VFM serves as the teacher for context features to enhance spatial consistency.