Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity

Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida, Toshimitsu Uesaka, Hiromi Wakaki, Yuki Mitsufuji

TL;DR

The paper tackles the problem that CLIP-style models do not fully exploit the theoretically optimal similarity PMI between modalities. It introduces KME-CLIP, which embeds image and text representations into a reproducing kernel Hilbert space and defines the similarity as the log of the RKHS inner product, leveraging a linear structure in exp(PMI). The authors prove that, under mild assumptions, exp(PMI) can be approximated arbitrarily well by kernel mean embeddings as the point-set size grows, and they demonstrate that CLIP can fail to approximate PMI in certain data regimes while KME-CLIP succeeds. Empirically, KME-CLIP improves retrieval and zero-shot classification across multiple benchmarks and remains competitive under ablations reducing the point-set size, supporting its practical viability. Overall, the work provides a theoretically grounded, scalable refinement to cross-modal similarity that improves alignment and downstream performance.

Abstract

In this study, we propose an enhancement to the similarity computation mechanism in multi-modal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space. We theoretically prove that our method can approximate PMI with arbitrary accuracy and empirically demonstrate that our approach overall outperforms the standard CLIP formulation across several retrieval and classification tasks.

Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity

TL;DR

The paper tackles the problem that CLIP-style models do not fully exploit the theoretically optimal similarity PMI between modalities. It introduces KME-CLIP, which embeds image and text representations into a reproducing kernel Hilbert space and defines the similarity as the log of the RKHS inner product, leveraging a linear structure in exp(PMI). The authors prove that, under mild assumptions, exp(PMI) can be approximated arbitrarily well by kernel mean embeddings as the point-set size grows, and they demonstrate that CLIP can fail to approximate PMI in certain data regimes while KME-CLIP succeeds. Empirically, KME-CLIP improves retrieval and zero-shot classification across multiple benchmarks and remains competitive under ablations reducing the point-set size, supporting its practical viability. Overall, the work provides a theoretically grounded, scalable refinement to cross-modal similarity that improves alignment and downstream performance.

Abstract

In this study, we propose an enhancement to the similarity computation mechanism in multi-modal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space. We theoretically prove that our method can approximate PMI with arbitrary accuracy and empirically demonstrate that our approach overall outperforms the standard CLIP formulation across several retrieval and classification tasks.
Paper Structure (40 sections, 8 theorems, 50 equations, 1 figure, 7 tables)

This paper contains 40 sections, 8 theorems, 50 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Contrastive language-image pre-training (CLIP)
  4. Pointwise mutual information as best similarity metric
  5. Proposed method
  6. Key insight: linear structure in $\exp(\text{PMI})$
  7. Proposed method: KME-CLIP
  8. Implementation
  9. Comparison with other existing methods
  10. Comparison with CLIP
  11. Comparison with WPSE
  12. Theory
  13. Theoretical guarantee for KME-CLIP
  14. Theoretical comparison of CLIP and KME-CLIP under low modality gap
  15. Experiments
  16. ...and 25 more sections

Key Result

Theorem 1

$L_S$ in exp_of_loss attains its minimum if we have $S(x,y)=\text{\rm PMI}(x, y) +\mathrm{const}$ for all $x\in\mathcal{X}$ and $y\in\mathcal{Y}$.

Figures (1)

  • Figure 1: Overview of our proposed method. We project the image and text embedding into RKHS and calculate similarity by the logarithm of the inner product in this space.

Theorems & Definitions (16)

  • Definition 1: Pointwise mutual information
  • Theorem 1: Proposition1, zhang2023deep
  • Proposition 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • proof
  • Lemma 8
  • ...and 6 more