Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing

TL;DR

EDIR introduces a fine-grained benchmark for Composed Image Retrieval (CIR) built via a controllable image-editing pipeline. It defines a hierarchical taxonomy of five categories and fifteen subcategories, and generates 5,000 queries over a gallery of 178,645 images. Comprehensive evaluation of 13 multimodal embedding models reveals persistent gaps in fine-grained editing, negation, and complex compositional reasoning, alongside biases in existing benchmarks. An in-domain training experiment shows substantial gains on data-solvable categories, while highlighting intrinsic architectural limits on others. Overall, EDIR provides a diagnostic tool to drive development of genuinely compositional and less biased CIR systems.

Abstract

Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.

Figures (13)

• Figure 1: (a) Query category distribution in existing benchmarks, re-categorized using our taxonomy. (b) The balanced distribution of EDIR across five main categories and fifteen subcategories. (c) Example queries illustrating the fine-grained nature of each subcategory. Left is the source image, right is the target image, and the below text is the CIR query text.
• Figure 2: Overview of our data synthesis pipeline: (1) Seed Image Selection: Unsuitable images are filtered from a large pool to select high-quality source images. (2) Triplet Generation: For each source image, multiple edit instructions are generated and applied to create ⟨source image, edit instruction, target image⟩ triplets. (3) Query Formulation: The edit instructions are automatically rewritten into natural language CIR queries.
• Figure 3: Performance correlation of MLLM-based models between EDIR and prior CIR benchmarks.
• Figure 4: Average performance of MLLM-based models across CIR benchmarks.
• Figure 5: Example of Error Type: Failure in Handling Negation