DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks

TL;DR

DetailCLIP addresses the gap in vision-language pretraining for fine-grained tasks by integrating patch-level self-distillation, pixel-level reconstruction, and attention-guided token filtering within a teacher-student EMA framework. It combines global CLIP alignment with fine-grained, patch-wise and pixel-wise objectives to preserve detailed visual features while maintaining semantic support from text. The approach yields state-of-the-art segmentation results on ADE20K and strong COCO detections, along with competitive zero-shot text-image retrieval and classification, demonstrating superior generalization to detail-oriented tasks. This work advances robust, detail-preserving vision-language representations with practical impact on segmentation, detection, and cross-modal understanding in resource-efficient settings.

Abstract

In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like segmentation. While CLIP and its variants excel in the global alignment of image and text representations, they often struggle to capture the fine-grained details necessary for precise segmentation. To overcome these challenges, we propose a novel framework that employs patch-level comparison of self-distillation and pixel-level reconstruction losses, enhanced with an attention-based token removal mechanism. This approach selectively retains semantically relevant tokens, enabling the model to focus on the image's critical regions aligned with the specific functions of our model, including textual information processing, patch comparison, and image reconstruction, ensuring that the model learns high-level semantics and detailed visual features. Our experiments demonstrate that DetailCLIP surpasses existing CLIP-based and traditional self-supervised learning (SSL) models in segmentation accuracy and exhibits superior generalization across diverse datasets. DetailCLIP represents a significant advancement in vision-language modeling, offering a robust solution for tasks that demand high-level semantic understanding and detailed feature extraction. https://github.com/KishoreP1/DetailCLIP.

Figures (7)

• Figure 1: Using a teacher-student architecture, the teacher processes two views of an input image to generate attention values, guiding token removal in the student model. The student reconstructs the image with a vision decoder while optimizing classification loss ($\mathcal{L}_{CLS}$), patch contrastive loss ($\mathcal{L}_{Patch}$), and reconstruction loss ($\mathcal{L}_{Rec}$). The CLIP loss ($\mathcal{L}_{CLIP}$) ensures alignment between vision and text encoders, enhancing segmentation. In the figure, $g$ is the vision encoder, $h$ the projection head, $d$ the decoder, and $e$ the text encoder.
• Figure 2: Visual comparison of segmentation results using the Linear decoder across different models. The comparison includes SLIP, Attentive Mask CLIP, and DetailCLIP at 25 epochs.
• Figure 3: Illustration of the token removal mechanism, which retains semantically significant tokens to focus on critical input aspects while discarding unnecessary details.
• Figure 4: Visual comparison of segmentation results using the UPerNet decoder across different models. The comparison includes SLIP, A-CLIP, DetailCLIP at 25 epochs, and DetailCLIP at 50 epochs.
• Figure 5: Visual comparison of segmentation results using the Linear decoder across different models. The comparison includes SLIP, A-CLIP, and DetailCLIP at 25 epochs.