Data or Language Supervision: What Makes CLIP Better than DINO?

TL;DR

This work systematically disentangles the effects of language supervision and data scale on vision encoders for vision-language models. By training CLIP and DINO under identical architecture, data, and optimization, and analyzing embeddings and downstream VLM performance, it shows CLIP's representations are more semantically structured and text-aware, while DINO emphasizes low-level cues. When integrated into VLMs, CLIP excels at text-heavy tasks like OCR in VQA, whereas DINO can match CLIP on many tasks but lags in text-centric scenarios. Furthermore, exploring alternative language supervision objectives or pretrained language priors yields limited gains, highlighting the primacy of supervision type in combination with data scale for shaping vision encoder quality.

Abstract

CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings -- using the same architecture, dataset, and training configuration -- achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.

Figures (4)

• Figure 1: CLIP and DINO represent two predominant paradigms of vision encoders, differing in two key aspects: (1) CLIP is trained with language supervision, whereas DINO uses image-only self-supervision; (2) CLIP and its variant SigLIP are trained on datasets that are up to 100 times larger than those used for DINO. These differences make it difficult to disentangle whether CLIP’s superior performance in vision-language models stems from its training objective or the scale of its training data.
• Figure 2: Embedding analysis of CLIP and DINO. The top two image pairs exhibit high cosine similarity according to CLIP but low similarity under DINO, suggesting that CLIP is more attuned to high-level semantics such as object categories and embedded text. In contrast, the bottom pairs show the opposite pattern, indicating that DINO is more sensitive to low-level features like object colors and visual styles.
• Figure 3: VLM analysis of CLIP and DINO. We integrate the controlled CLIP and DINO encoders into LLaVA-1.5 and evaluate on 20 subsets of the VMCBench benchmark. Results show that LLaVA-CLIP significantly outperforms LLaVA-DINO on OCR tasks by 7.5%, while their performance is largely comparable on other tasks.
• Figure 4: Training loss curves for the controlled CLIP (top) and DINO (bottom) models.