Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning

TL;DR

This work tackles the interpretability challenge of CLIP by integrating sparsity directly into training rather than applying post-hoc sparsification. It introduces Sparse CLIP, which imposes non-negativity via ReLU and substantially expands the representation dimensionality, underpinned by a dictionary-learning perspective $A \approx UV^T$ with $U,V \ge 0$. The approach yields sparse, multimodal representations that retain downstream performance and enable straightforward semantic alignment of concepts, as well as insights into how cross-modal knowledge emerges during training. The authors demonstrate practical utility by deploying a vision-language model with interpretable steering, arguing that interpretability and performance can be co-optimized as a core design principle rather than a trade-off. This suggests a promising direction for designing future multimodal models with built-in interpretability without sacrificing accuracy.

Abstract

Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in vision-language representation learning, powering diverse downstream tasks and serving as the default vision backbone in multimodal large language models (MLLMs). Despite its success, CLIP's dense and opaque latent representations pose significant interpretability challenges. A common assumption is that interpretability and performance are in tension: enforcing sparsity during training degrades accuracy, motivating recent post-hoc approaches such as Sparse Autoencoders (SAEs). However, these post-hoc approaches often suffer from degraded downstream performance and loss of CLIP's inherent multimodal capabilities, with most learned features remaining unimodal. We propose a simple yet effective approach that integrates sparsity directly into CLIP training, yielding representations that are both interpretable and performant. Compared to SAEs, our Sparse CLIP representations preserve strong downstream task performance, achieve superior interpretability, and retain multimodal capabilities. We show that multimodal sparse features enable straightforward semantic concept alignment and reveal training dynamics of how cross-modal knowledge emerges. Finally, as a proof of concept, we train a vision-language model on sparse CLIP representations that enables interpretable, vision-based steering capabilities. Our findings challenge conventional wisdom that interpretability requires sacrificing accuracy and demonstrate that interpretability and performance can be co-optimized, offering a promising design principle for future models.

Figures (15)

• Figure 1: Comparison between Sparse CLIP and two existing methods, CLIP and SAE.
• Figure 2: Small-scale ablation study. We evaluated different Sparse CLIP design choices by training ViT-B/32 models from scratch on a 15M sample dataset.
• Figure 3: Quantitative interpretability evaluations of Sparse CLIP embeddings.
• Figure 4: Examples features of ViT-L/14 Sparse+. The top activated words and images for these multimodal features are highly correlated, so we can directly name their visual concept based on text.
• Figure 5: Evolution of "dog rose" feature. Top activated words and images at different stage of training. Activations values are normalized.