Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric
Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chieh-Hsin Lai, Naoki Murata, Yuki Mitsufuji
TL;DR
This paper addresses the limitation of single-vector representations in multimodal contrastive learning by introducing Weighted Point Set Embedding (WPSE), where each concept is represented as a weighted set of vectors and similarities are computed via kernelized pairwise interactions. It establishes a theoretical connection that the optimal symmetric InfoNCE similarity corresponds to the pointwise mutual information (PMI) and shows that, under this optimal similarity, a linear classifier on top of the learned WPSE representations can achieve near-optimal downstream performance. The authors prove a finite-sample excess-risk bound that decomposes into KL-divergence terms, analyze the impact of deviations from PMI, and demonstrate that a nonlinear kernel over weighted point sets can approximate PMI arbitrarily well with a mild generation assumption. Empirically, WPSE pretrained on large text–image datasets with random Fourier feature–based kernel approximations yields improved zero-shot transfer and linear classification results over CLIP baselines on multiple benchmarks, validating both the theory and practical utility of the approach.
Abstract
In typical multimodal contrastive learning, such as CLIP, encoders produce one point in the latent representation space for each input. However, one-point representation has difficulty in capturing the relationship and the similarity structure of a huge amount of instances in the real world. For richer classes of the similarity, we propose the use of weighted point sets, namely, sets of pairs of weight and vector, as representations of instances. In this work, we theoretically show the benefit of our proposed method through a new understanding of the contrastive loss of CLIP, which we call symmetric InfoNCE. We clarify that the optimal similarity that minimizes symmetric InfoNCE is the pointwise mutual information, and show an upper bound of excess risk on downstream classification tasks of representations that achieve the optimal similarity. In addition, we show that our proposed similarity based on weighted point sets consistently achieves the optimal similarity. To verify the effectiveness of our proposed method, we demonstrate pretraining of text-image representation models and classification tasks on common benchmarks.