Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP

TL;DR

This work tackles the poor visio-linguistic reasoning of CLIP by introducing SDS-CLIP, a data- and parameter-efficient distillation that regularizes CLIP with a denoising diffusion score from a text-to-image model (Stable Diffusion). By freezing the diffusion UNet and only training a small mapper plus LayerNorm parameters, the method distills reasoning capabilities into CLIP using a loss $L_{\text{total}}=L_{\text{CLIP}}+\lambda L_{\text{SDS}}$ with $L_{\text{SDS}}=\mathbb{E}_{t,\epsilon}[\| \epsilon_{\theta}(h_{w}(f_{\phi}(x)),t,c)-\epsilon\|^{2}]$. Empirically, SDS-CLIP yields consistent improvements on Winoground ($1.5\%-7\%$) and ARO ($1\%-3\%$) while largely preserving zero-shot performance, and demonstrates compatibility with multiple CLIP backbones and OpenCLIP. The work highlights how a carefully designed diffusion-based regularizer can transfer compositional reasoning from generative models to discriminative vision-language models, enabling stronger visio-linguistic understanding with minimal computational overhead compared to full diffusion inference.

Abstract

Image-text contrastive models like CLIP have wide applications in zero-shot classification, image-text retrieval, and transfer learning. However, they often struggle on compositional visio-linguistic tasks (e.g., attribute-binding or object-relationships) where their performance is no better than random chance. To address this, we introduce SDS-CLIP, a lightweight and sample-efficient distillation method to enhance CLIP's compositional visio-linguistic reasoning. Our approach fine-tunes CLIP using a distillation objective borrowed from large text-to-image generative models like Stable-Diffusion, which are known for their strong visio-linguistic reasoning abilities. On the challenging Winoground benchmark, SDS-CLIP improves the visio-linguistic performance of various CLIP models by up to 7%, while on the ARO dataset, it boosts performance by up to 3%. This work underscores the potential of well-designed distillation objectives from generative models to enhance contrastive image-text models with improved visio-linguistic reasoning capabilities.

Figures (2)

• Figure 1: CLIP variants underperform on Winoground, a visio-linguistic reasoning benchmark, compared to Diffusion Score from Stable Diffusion. The diffusion score is computed from Stable Diffusion's loss function. Note that Diffusion Score takes $~18 \times$ more time than CLIP variants for inference (using 50 samplings during diffusion score computation).
• Figure 2: Our fine-tuning method does not harm the zero-shot abilities of CLIP. In fact for certain downstream datasets (e.g., ImageNet, CIFAR-10, MNIST, Aircraft) -- we observe an improvement in the zero-shot performance between $1\%-8\%$ for ViT-B/16. For other CLIP models, we find no drop in zero-shot performance.