Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Method to Capturing Compositional Knowledge in Linguistic Space

Jiahe Wan

TL;DR

This work tackles zero-shot compositional understanding in vision–language models by introducing YUKINO, a method that maps images into pseudo-tokens within the CLIP embedding space via textual inversion. It combines an optimization-based inversion phase (OTI) with a distillation phase to train a fast inverter, Theta, and introduces a no-logic regularization scheme to stabilize token interactions. Through two-phase training on unlabeled data, YUKINO achieves state-of-the-art results on SugarCREPE and strong gains on Winoground, demonstrating improved compositionality without expensive labeled data. The approach yields faster inference while preserving high-quality image–caption alignment, offering practical benefits for retrieval tasks and robust cross-modal understanding in real-world applications.

Abstract

Compositional understanding allows visual language models to interpret complex relationships between objects, attributes, and relations in images and text. However, most existing methods often rely on hard negative examples and fine-tuning, which can overestimate improvements and are limited by the difficulty of obtaining hard negatives. In this work, we introduce Zero-Shot Compositional Understanding (ZS-CU), a novel task that enhances compositional understanding without requiring hard negative training data. We propose YUKINO (Yielded Compositional Understanding Knowledge via Textual Inversion with NO), which uses textual inversion to map unlabeled images to pseudo-tokens in a pre-trained CLIP model. We propose introducing "no" logical regularization to address the issue of token interaction in inversion. Additionally, we suggest using knowledge distillation to reduce the time complexity of textual inversion. Experimental results show that YUKINO outperforms the existing multi-modal SOTA models by over 8% on the SugarCREPE benchmark, and also achieves significant improvements in image retrieval tasks.

A New Method to Capturing Compositional Knowledge in Linguistic Space

TL;DR

This work tackles zero-shot compositional understanding in vision–language models by introducing YUKINO, a method that maps images into pseudo-tokens within the CLIP embedding space via textual inversion. It combines an optimization-based inversion phase (OTI) with a distillation phase to train a fast inverter, Theta, and introduces a no-logic regularization scheme to stabilize token interactions. Through two-phase training on unlabeled data, YUKINO achieves state-of-the-art results on SugarCREPE and strong gains on Winoground, demonstrating improved compositionality without expensive labeled data. The approach yields faster inference while preserving high-quality image–caption alignment, offering practical benefits for retrieval tasks and robust cross-modal understanding in real-world applications.

Abstract

Compositional understanding allows visual language models to interpret complex relationships between objects, attributes, and relations in images and text. However, most existing methods often rely on hard negative examples and fine-tuning, which can overestimate improvements and are limited by the difficulty of obtaining hard negatives. In this work, we introduce Zero-Shot Compositional Understanding (ZS-CU), a novel task that enhances compositional understanding without requiring hard negative training data. We propose YUKINO (Yielded Compositional Understanding Knowledge via Textual Inversion with NO), which uses textual inversion to map unlabeled images to pseudo-tokens in a pre-trained CLIP model. We propose introducing "no" logical regularization to address the issue of token interaction in inversion. Additionally, we suggest using knowledge distillation to reduce the time complexity of textual inversion. Experimental results show that YUKINO outperforms the existing multi-modal SOTA models by over 8% on the SugarCREPE benchmark, and also achieves significant improvements in image retrieval tasks.

Paper Structure

This paper contains 33 sections, 10 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Compositional Understanding
  4. Textual Inversion
  5. Knowledge Distillation
  6. Methodology
  7. Overview
  8. Optimization-based Textual Inversion (OTI)
  9. Pre-training of textual inversion network
  10. Experiments
  11. Datasets and Evaluation Metrics
  12. Pretraining Datasets
  13. Downstream Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 18 more sections

Figures (4)

  • Figure 1: YUKINO introduces a simple enhancement to CLIP that significantly improves its compositional understanding. (a) CLIP and our method YUKINO give similarity scores for captions paired with aligned and unaligned images. (b) Workflow of our proposed method YUKINO under image-to-text retrieval.
  • Figure 2: Overview of YUKINO. (a) Optimization-based Textual Inversion: we generate a pseudo-token $v_{*}$ from an image I(left part). (b) Pre-training of textual inversion network: we train a network on unlabeled images to gain the ability to quickly invert arbitrary images(right part)
  • Figure 3: Similarity density of CLIP, Neg-CLIP, Structure-CLIP and YUKINO. (a), (b), (c) and (d) show the similarity density of image0 with all captions and caption0 with all images in Winoground for the 4 models, respectively.
  • Figure 4: Predictions of different approaches. The words in red and blue are difference words. We compare our YUKINO with CLIP to calculate CLIP scores (i.e., semantic similarity) between the image and captions.