SILC: Improving Vision Language Pretraining with Self-Distillation

TL;DR

SILC addresses the limitation of contrastive vision-language pretraining in learning dense, locally grounded features by adding a local-to-global self-distillation objective. The framework trains a two-tower VLM with an EMA teacher that distills local image semantics from partial views to align with global semantics and language, guided by two complementary objectives. Empirically, SILC achieves new state-of-the-art performance across zero-shot, few-shot, retrieval, segmentation (zero-shot and open vocabulary), open vocabulary detection, captioning, and VQA, on WebLI-based pretraining. This approach yields robust, locally aware representations that improve both image-level and pixel-level vision-language tasks, marking a significant advance in open-vocabulary vision systems.

Abstract

Image-Text pretraining on web-scale image caption datasets has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants. Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities. However, the contrastive objective used by these models only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks. In this work, we introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation, while also providing improvements on image-level tasks such as classification and retrieval. SILC models sets a new state of the art for zero-shot classification, few shot classification, image and text retrieval, zero-shot segmentation, and open vocabulary segmentation. We further show that SILC features greatly benefit open vocabulary detection, captioning and visual question answering.

Figures (5)

• Figure 1: SILC improves image-text contrastive learning with the addition of local-to-global correspondence learning by self-distillation. As a result, SILC models learn more locally aware visual features that are also grounded in language. SILC models offer significant improvements over CLIP (WebLI) and SigLIP over a wide variety of computer vision tasks including classification, segmentation, detection, captioning, VQA and retrieval.
• Figure 2: SILC is a two-tower transformer based VLM. The first component of our training objective uses a global view of an image covering a large area and its paired caption to optimise a batch-wise contrastive loss for images and texts. The second component of our training objective enforces local-to-global consistency by self-distillation between the main model (the student) and an Exponential Moving Average (EMA)-based teacher. This local-to-global correspondence additionally allows the model to learn good visual features. Together the two objectives allow the model to excel at both traditional VLM tasks as well as tasks that require local understanding like segmentation and detection.
• Figure 3: Qualitative results on zero-shot segmentation show that SILC-C achieves significant improvements over CLIP (WebLI). SILC-C produces less noisy segmentation and better distinguishes semantic classes. This semantic segmentation emerges without any segmentation supervision.
• Figure 4: Comparing Open Vocabulary Semantic Segmentation performance, we observe that SILC models improve over CLIP by significant margins on all unseen test sets. SILC particularly improves the performance for challenging test sets with large vocabularies. SILC-L/16 even outperforms the much larger CLIP-G/14. All models are trained on COCO-Stuff.
• Figure 5: Comparing qualitative examples for open vocabulary segmentation, we observe that SILC w/ CAT-Seg better distinguishes semantically similar classes such as field/grass, runway/road, grandstand/chair and sand/water than CLIP.