Whitened CLIP as a Likelihood Surrogate of Images and Captions

TL;DR

The paper tackles the problem of estimating likelihoods for images and captions by introducing Whitened CLIP (W-CLIP), a training-free whitening of CLIP embeddings that renders the transformed features approximately IID standard normal. The log-likelihood is then approximated by the squared Euclidean norm in the whitened space, via $\\ell(x) = -\\frac{1}{2}\left(d\\log(2\\pi) + \\|x\\|^2\right)$, leveraging the chi-distribution of norms. The authors validate normality with Anderson-Darling and D’Agostino-Pearson tests, demonstrate uniformity in cosine similarities, and explore practical uses such as artifact detection, domain-shift assessment, data generalization, and a full-circle SLERP for image interpolation/extrapolation. This work provides a fast, principled likelihood surrogate rooted in CLIP semantics, with potential impact on generation evaluation, detection, and manipulation tasks across vision-language systems. The accompanying code enables reproducibility and further exploration of likelihood-based analysis in CLIP spaces.

Abstract

Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce \textit{Whitened CLIP}, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embeddings statistics can be well approximated as a standard normal distribution, thus, the log-likelihood is estimated simply by the square Euclidean norm in the whitened embedding space. The whitening procedure is completely training-free and performed using a pre-computed whitening matrix, hence, is very fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions.

Figures (26)

• Figure 1: Raw, centered and whitened CLIP geometry. The whitened CLIP space is isotropic, transforming the original ellipsoid shaped space into an hypersphere. In this space, the embedding norm reflects likelihood level. Higher norms correspond to lower probabilities.
• Figure 2: Log-likelihood of real and generated images with artifacts. Real images of a hand and a dog (left) and three similar AI generated images with artifacts. Real images have higher log-likelihood than generated images with artifacts.
• Figure 3: Norm histograms of ImageNet variations. Top left: ImageNet-A, comprising of natural adversarial examples, closely aligns with clean ImageNet due to their natural origins. Top right: ImageNet-C histograms under varying impulse noise levels of severity display significantly larger norms than clean ImageNet, indicating distributional deviations. Bottom: ImageNet-R comparison shows that different styles cause varying likelihood shifts, with graffiti closest to real images and video game renditions exhibiting the largest shifts.
• Figure 4: Raw CLIP and W-CLIP analytic comparison. The covariance matrices of raw CLIP (a) and W-CLIP (b) demonstrate the effectiveness of the whitening transformation in achieving unit variance and zero correlation among features. Histograms of four CLIP features (c) vary in mean and variance, whereas four W-CLIP features (d) exhibit zero mean and unit variance. Cosine similarity histograms for all image pairs (e) across raw, centered, and W-CLIP embeddings reveal that W-CLIP's cosine similarity is concentrated around zero, indicating significantly improved uniformity compared to the centered and raw CLIP spaces.
• Figure 5: Likelihood variation for different levels of details. The original MS-COCO caption is marked with an arrow, with deviations underlined. Left: Removing details, such as character names or locations, increases likelihood. Right: Adding specificity, such as replacing "woman" with "bride" or "Jenny", decreases likelihood.