Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Probabilistic Image-Text Representations

Sanghyuk Chun

TL;DR

This work tackles the inherent ambiguity in image-text matching by modeling images and captions as probabilistic embeddings and introducing a closed-form probabilistic distance (CSD) to replace Monte Carlo matching. It adds two soft-label strategies, pseudo-positives (PP) and mixed sample data augmentation (MSDA), and uses a CLIP-like architecture with a focused uncertainty head to improve robustness against false negatives and noisy correspondences. The approach, PCME++, demonstrates superior or competitive retrieval performance across COCO, ECCV Caption, and CxC benchmarks, scales well with larger backbones, and enables practical uses such as large-scale retrieval and uncertainty-based zero-shot prompt-filtering. By providing interpretable uncertainty measures, PCME++ offers not only better accuracy but also controllability and potential bias mitigation in vision-language understanding and downstream tasks.

Abstract

Image-Text Matching (ITM) task, a fundamental vision-language (VL) task, suffers from the inherent ambiguity arising from multiplicity and imperfect annotations. Deterministic functions are not sufficiently powerful to capture ambiguity, prompting the exploration of probabilistic embeddings to tackle the challenge. However, the existing probabilistic ITM approach encounters two key shortcomings; the burden of heavy computations due to the Monte Carlo approximation, and the loss saturation issue in the face of abundant false negatives. To overcome the issues, this paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) by introducing a new probabilistic distance with a closed-form solution. In addition, two optimization techniques are proposed to enhance PCME++ further: first, the incorporation of pseudo-positives to prevent the negative effect under massive false negatives; second, mixed sample data augmentation for probabilistic matching. Experimental results on MS-COCO Caption and two extended benchmarks, CxC and ECCV Caption, demonstrate the effectiveness of PCME++ compared to state-of-the-art ITM methods. The robustness of PCME++ is also evaluated under noisy image-text correspondences. In addition, the potential applicability of PCME++ in automatic prompt-filtering for zero-shot classification is shown. The code is available at https://github.com/naver-ai/pcmepp

Improved Probabilistic Image-Text Representations

TL;DR

This work tackles the inherent ambiguity in image-text matching by modeling images and captions as probabilistic embeddings and introducing a closed-form probabilistic distance (CSD) to replace Monte Carlo matching. It adds two soft-label strategies, pseudo-positives (PP) and mixed sample data augmentation (MSDA), and uses a CLIP-like architecture with a focused uncertainty head to improve robustness against false negatives and noisy correspondences. The approach, PCME++, demonstrates superior or competitive retrieval performance across COCO, ECCV Caption, and CxC benchmarks, scales well with larger backbones, and enables practical uses such as large-scale retrieval and uncertainty-based zero-shot prompt-filtering. By providing interpretable uncertainty measures, PCME++ offers not only better accuracy but also controllability and potential bias mitigation in vision-language understanding and downstream tasks.

Abstract

Image-Text Matching (ITM) task, a fundamental vision-language (VL) task, suffers from the inherent ambiguity arising from multiplicity and imperfect annotations. Deterministic functions are not sufficiently powerful to capture ambiguity, prompting the exploration of probabilistic embeddings to tackle the challenge. However, the existing probabilistic ITM approach encounters two key shortcomings; the burden of heavy computations due to the Monte Carlo approximation, and the loss saturation issue in the face of abundant false negatives. To overcome the issues, this paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) by introducing a new probabilistic distance with a closed-form solution. In addition, two optimization techniques are proposed to enhance PCME++ further: first, the incorporation of pseudo-positives to prevent the negative effect under massive false negatives; second, mixed sample data augmentation for probabilistic matching. Experimental results on MS-COCO Caption and two extended benchmarks, CxC and ECCV Caption, demonstrate the effectiveness of PCME++ compared to state-of-the-art ITM methods. The robustness of PCME++ is also evaluated under noisy image-text correspondences. In addition, the potential applicability of PCME++ in automatic prompt-filtering for zero-shot classification is shown. The code is available at https://github.com/naver-ai/pcmepp
Paper Structure (47 sections, 3 theorems, 8 equations, 15 figures, 17 tables)

This paper contains 47 sections, 3 theorems, 8 equations, 15 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Improved Probabilistic Cross-Modal Embeddings (PCME++)
  4. Problem definition: Ambiguity of ITM datasets
  5. Probabilistic contrastive learning
  6. Pseudo-positives (PP) for handling numerous false negatives
  7. Mixed Sample Data Augmentation (MSDA) for probabilistic matching
  8. Architecture
  9. Experiments
  10. Experimental protocol
  11. Comparison methods.
  12. Training details and model selection.
  13. COCO ITM results
  14. Main results.
  15. Noisy correspondence.
  16. ...and 32 more sections

Key Result

Lemma 1

Let $X$ and $Y$ be independent normally distributed random variables where $X \sim \mathcal{N}(\mu_X, \Sigma_X)$ and $Y \sim \mathcal{N}(\mu_Y, \Sigma_Y)$. Then, the subtraction between $X$ and $Y$ is another normal distribution, i.e., $(X - Y) \sim \mathcal{N} (\mu_X - \mu_Y, \Sigma_X + \Sigma_Y).$

Figures (15)

  • Figure 1: Inherent ambiguity of ITM. We assume that the deterministic textual embeddings are mapped to the same point $z_t^\prime$, i.e., $z_t^1 \approx z_t^2 \approx z_t^3 \approx z_t^\prime$, as well as the probabilistic textual embeddings $\mathbf{Z}_t\xspace^1 \approx \ldots \approx \mathbf{Z}_t\xspace^\prime$.
  • Figure 2: Architecture overview. We use the same visual and textual backbones as CLIP. Each modality encoder encodes $\ell_2$-normalized mean vector $\mu$ and the variance vector $\log \sigma^2$, followed by Generalized Pooling Operator (GPO) chen2021vseinfty, to represent a normally distributed random variable $\mathbf{Z} \sim \mathcal{N} (\mu, \sigma^2)$.
  • Figure 3: $\|\sigma^2\|_1$ vs. R@1.
  • Figure 4: 2D t-SNE visualization of learned embeddings by PCME++ and VSE$\infty$. The area of probabilistic embeddings denotes the uncertainty of each embedding, i.e., a more uncertain sample has a larger area.
  • Figure A.1: Toy results for HNM & SUM VSE, Wesserstien distance & PCME++. The full animation can be found in https://naver-ai.github.io/pcmepp/.
  • ...and 10 more figures

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Proposition 1
  • proof