Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PowerCLIP: Powerset Alignment for Contrastive Pre-Training

Masaki Kawamura, Nakamasa Inoue, Rintaro Yanagi, Hirokatsu Kataoka, Rio Yokota

TL;DR

PowerCLIP addresses the challenge of modeling compositionality in vision-language pre-training by exhaustively aligning powersets of image regions with textual parse-tree phrases. It introduces Non-Linear Aggregators to approximate the exponential powerset computations with linear-time complexity, backed by theoretical guarantees. Empirical results show PowerCLIP achieves state-of-the-art performance across 28 benchmarks, including zero-shot classification, retrieval, robustness, and compositionality, outperforming strong global and local alignment baselines. The approach demonstrates robust, fine-grained text-to-image grounding and grounding-to-text reasoning, with practical implications for more compositional multimodal understanding.

Abstract

Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Our code will be made publicly available.

PowerCLIP: Powerset Alignment for Contrastive Pre-Training

TL;DR

PowerCLIP addresses the challenge of modeling compositionality in vision-language pre-training by exhaustively aligning powersets of image regions with textual parse-tree phrases. It introduces Non-Linear Aggregators to approximate the exponential powerset computations with linear-time complexity, backed by theoretical guarantees. Empirical results show PowerCLIP achieves state-of-the-art performance across 28 benchmarks, including zero-shot classification, retrieval, robustness, and compositionality, outperforming strong global and local alignment baselines. The approach demonstrates robust, fine-grained text-to-image grounding and grounding-to-text reasoning, with practical implications for more compositional multimodal understanding.

Abstract

Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Our code will be made publicly available.

Paper Structure

This paper contains 64 sections, 53 equations, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contrastive Pre-training.
  4. Textual Approaches.
  5. Multimodal Approaches.
  6. Method
  7. Overview
  8. Problem Setting and Notation.
  9. Architecture.
  10. Region Embedding Extraction
  11. Region Masks.
  12. Region Embeddings.
  13. Phrase Embedding Extraction
  14. Parse Trees.
  15. Token Masks.
  16. ...and 49 more sections

Figures (11)

  • Figure 1: Overview of PowerCLIP. (a) CLIP aligns images and sentences globally. (b) PowerCLIP explores all combinations of image regions (i.e. , powerset) and aligns them with textual phrases.
  • Figure 2: Performance comparison between PowerCLIP and the best-performing method among seven state-of-the-art approaches (CLIP, FLIP, A-CLIP, E-CLIP, C-PGS, FILIP, and SPARC). Performance improvements are highlighted in red.
  • Figure 3: Overview of the powerset alignment strategy for PowerCLIP. (a) Region embeddings are extracted for each subset $A$ of region masks in $\mathcal{M}$. (b) Phrase embeddings are extracted for each node $B$ in the parse tree $\mathcal{T}$. (c) Powerset alignment minimizes the triplet loss defined based on the bidirectional similarity: region-set-to-tree (R2T) and vice versa (T2R).
  • Figure 4: Non-Linear Aggregator (NLA). Each layer applies aggregation followed by activation.
  • Figure 5: Approximation accuracy evaluation. Top: Comparison between exact and approximated losses for $\tau = \{0.1, 0.01, 0.001\}$ and $\alpha \in \{0.00, 0.25, 0.50, 0.75, 1.00\}$. Bottom: Pearson correlation $r$ between exact and approximated losses.
  • ...and 6 more figures

Theorems & Definitions (5)

  • proof
  • proof
  • proof
  • proof
  • proof