Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training-free Conditional Image Embedding Framework Leveraging Large Vision Language Models

Masayuki Kawarada, Kosuke Yamada, Antonio Tejero-de-Pablos, Naoto Inoue

TL;DR

DIOR introduces a training-free approach to conditional image embeddings by leveraging large vision-language models (LVLMs) and prompting them to describe images in one word conditioned on a given attribute. The method extracts the LVLM’s last prompt-token hidden state as the conditional embedding, enabling targeted image similarity without task-specific priors or fine-tuning. Across LanZ-DML, Style Similarity, and GeneCIS, DIOR demonstrates competitive or superior performance against training-based baselines, with ablations clarifying the importance of layer choice, token extraction, and prompt design. While DIOR incurs higher inference costs due to LVLM size, caching and emerging efficient LVLMs mitigate this, presenting a practical, flexible pathway for cross-domain conditional image retrieval.

Abstract

Conditional image embeddings are feature representations that focus on specific aspects of an image indicated by a given textual condition (e.g., color, genre), which has been a challenging problem. Although recent vision foundation models, such as CLIP, offer rich representations of images, they are not designed to focus on a specified condition. In this paper, we propose DIOR, a method that leverages a large vision-language model (LVLM) to generate conditional image embeddings. DIOR is a training-free approach that prompts the LVLM to describe an image with a single word related to a given condition. The hidden state vector of the LVLM's last token is then extracted as the conditional image embedding. DIOR provides a versatile solution that can be applied to any image and condition without additional training or task-specific priors. Comprehensive experimental results on conditional image similarity tasks demonstrate that DIOR outperforms existing training-free baselines, including CLIP. Furthermore, DIOR achieves superior performance compared to methods that require additional training across multiple settings.

Training-free Conditional Image Embedding Framework Leveraging Large Vision Language Models

TL;DR

DIOR introduces a training-free approach to conditional image embeddings by leveraging large vision-language models (LVLMs) and prompting them to describe images in one word conditioned on a given attribute. The method extracts the LVLM’s last prompt-token hidden state as the conditional embedding, enabling targeted image similarity without task-specific priors or fine-tuning. Across LanZ-DML, Style Similarity, and GeneCIS, DIOR demonstrates competitive or superior performance against training-based baselines, with ablations clarifying the importance of layer choice, token extraction, and prompt design. While DIOR incurs higher inference costs due to LVLM size, caching and emerging efficient LVLMs mitigate this, presenting a practical, flexible pathway for cross-domain conditional image retrieval.

Abstract

Conditional image embeddings are feature representations that focus on specific aspects of an image indicated by a given textual condition (e.g., color, genre), which has been a challenging problem. Although recent vision foundation models, such as CLIP, offer rich representations of images, they are not designed to focus on a specified condition. In this paper, we propose DIOR, a method that leverages a large vision-language model (LVLM) to generate conditional image embeddings. DIOR is a training-free approach that prompts the LVLM to describe an image with a single word related to a given condition. The hidden state vector of the LVLM's last token is then extracted as the conditional image embedding. DIOR provides a versatile solution that can be applied to any image and condition without additional training or task-specific priors. Comprehensive experimental results on conditional image similarity tasks demonstrate that DIOR outperforms existing training-free baselines, including CLIP. Furthermore, DIOR achieves superior performance compared to methods that require additional training across multiple settings.
Paper Structure (39 sections, 4 equations, 7 figures, 12 tables)

This paper contains 39 sections, 4 equations, 7 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Embeddings
  4. Conditional Image Embeddings
  5. Large Vision-Language Models (LVLMs)
  6. Proposed Method
  7. Large Vision-Language Model
  8. DIOR Framework
  9. Experimental Setup
  10. Common Settings and Our Proposed Method
  11. Conditional Image Similarity Task.
  12. Common Baseline Method.
  13. Our Proposed Method.
  14. Benchmark-Specific Settings and Baselines
  15. LanZ-DML.
  16. ...and 24 more sections

Figures (7)

  • Figure 1: Images are projected into vector spaces based on given conditions, such as texture types or clothing categories. Our proposed method, DIOR, generates conditional embeddings by inputting the image and condition into a large vision-language model, without the need for additional training.
  • Figure 2: Overview of the DIOR framework. In our method, we first input the image along with a prompt describing the condition into a large vision-language model. The output from the final Transformer layer of the final token in the prompt is then used as the conditional image embedding.
  • Figure 3: Relationship between inference speed and average performance for each model on the LanZ-DML Benchmark. We use SentenceT5 as the text embedder for CapEmb. In InDiRECT and CLIP, "428M" and "18B" refer to CLIP L/14 and EVA-CLIP (18B), respectively.
  • Figure 4: Comparison of MAP@R for each layer's hidden state.
  • Figure 5: 2D t-SNE projections of the image embeddings by CLIP L/14 and DIOR on the Synthetic Cars and DeepFashion datasets. Each color indicates the class type in each condition.
  • ...and 2 more figures