Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIP

TL;DR

The paper tackles the problem of improving open-weight CLIP models without costly full pretraining. It introduces TuneCLIP, a two-stage self-supervised fine-tuning framework comprising Optimizer Statistics Recovery (OSR) and Hinged Global Contrastive Loss (HGCL) to address cold-start bias and false-negative penalization. The authors provide a theoretical basis for the cold-start issue, demonstrate improvements across multiple model scales and data regimes, and show that TuneCLIP yields gains on ImageNet, retrieval benchmarks, and the DataComp suite, notably enhancing open-weight baselines like OpenAI ViT-B/16 and SigLIP. The approach offers a practical, compute-efficient path to general-purpose multimodal representation refinement, establishing a new baseline for post-pretraining adaptation without labels.

Abstract

CLIP has become a cornerstone of multimodal representation learning, yet improving its performance typically requires a prohibitively costly process of training from scratch on billions of samples. We ask a different question: Can we improve the performance of open-weight CLIP models across various downstream tasks using only existing self-supervised datasets? Unlike supervised fine-tuning, which adapts a pretrained model to a single downstream task, our setting seeks to improve general performance across various tasks. However, as both our experiments and prior studies reveal, simply applying standard training protocols starting from an open-weight CLIP model often fails, leading to performance degradation. In this paper, we introduce TuneCLIP, a self-supervised fine-tuning framework that overcomes the performance degradation. TuneCLIP has two key components: (1) a warm-up stage of recovering optimization statistics to reduce cold-start bias, inspired by theoretical analysis, and (2) a fine-tuning stage of optimizing a new contrastive loss to mitigate the penalization on false negative pairs. Our extensive experiments show that TuneCLIP consistently improves performance across model architectures and scales. Notably, it elevates leading open-weight models like SigLIP (ViT-B/16), achieving gains of up to +2.5% on ImageNet and related out-of-distribution benchmarks, and +1.2% on the highly competitive DataComp benchmark, setting a new strong baseline for efficient post-pretraining adaptation.

Figures (11)

• Figure 1: Improvements delivered by TuneCLIP ($\bigstar$) over baseline models on complementary evaluation suites: large-scale DataComp Benchmark (38 datasets) & ImageNet’s 7 distributional variants.
• Figure 2: Zero-shot classification (%) performance on ImageNet-1k over 5 fine-tuning epochs for two OpenAI CLIP models (left: ViT-B/16, right: ViT-B/32). While FastCLIP and OpenCLIP show initial degradation and slow recovery, TuneCLIP maintains superior performance throughout fine-tuning.
• Figure 3: In supervised fine-tuning (red), OSR+GCL outperforms OSR+HGCL (TuneCLIP) because true negative labels justify separating negatives. In contrast, under self-supervised fine-tuning (blue), the absence of such labels makes OSR+HGCL more suitable, leading to improved retrieval performance on Flickr when fine-tuned with SSFT (see Appendix \ref{['app:benefits_hgcl']} for OpenAI CLIP & details).
• Figure 4: GCL improves classification but can degrades retrieval, whereas HGCL stabilizes retrieval while preserving overall classification gains.
• Figure 5: Effect of data scaling on TuneCLIP performance across models.

Theorems & Definitions (7)

• Theorem 4.1
• Theorem 4.2
• Lemma A.3: Lemma 8 from wang2022finite
• Lemma A.4
• Lemma A.5
• proof : proof of theorem \ref{['thm:main_thm_sox']}
• proof : proof of theorem \ref{['thm:stageI']}