InfoCIR: Multimedia Analysis for Composed Image Retrieval

TL;DR

It is argued that InfoCIR helps diagnose retrieval failures, guides prompt enhancement, and accelerates insight generation during model development, and accelerates insight generation during model development.

Abstract

Composed Image Retrieval (CIR) allows users to search for images by combining a reference image with a text prompt that describes desired modifications. While vision-language models like CLIP have popularized this task by embedding multiple modalities into a joint space, developers still lack tools that reveal how these multimodal prompts interact with embedding spaces and why small wording changes can dramatically alter the results. We present InfoCIR, a visual analytics system that closes this gap by coupling retrieval, explainability, and prompt engineering in a single, interactive dashboard. InfoCIR integrates a state-of-the-art CIR back-end (SEARLE arXiv:2303.15247) with a six-panel interface that (i) lets users compose image + text queries, (ii) projects the top-k results into a low-dimensional space using Uniform Manifold Approximation and Projection (UMAP) for spatial reasoning, (iii) overlays similarity-based saliency maps and gradient-derived token-attribution bars for local explanation, and (iv) employs an LLM-powered prompt enhancer that generates counterfactual variants and visualizes how these changes affect the ranking of user-selected target images. A modular architecture built on Plotly-Dash allows new models, datasets, and attribution methods to be plugged in with minimal effort. We argue that InfoCIR helps diagnose retrieval failures, guides prompt enhancement, and accelerates insight generation during model development. All source code allowing for a reproducible demo is available at https://github.com/giannhskp/InfoCIR.

Figures (4)

• Figure 1: The architecture links prompt composition, embedding-based retrieval, and interactive visualization across UI panels (A-F), enabling users to enhance prompts, interpret outputs, and derive insights. Following Wang et al. WANG2025100748, we integrate semantic fusion (CLIP space), visual integration (saliency and token overlays), and cross-view linking (synchronized views).
• Figure 2: Conceptual overview of the Class Projection Preservation Pipeline. ImageNet-R embeddings are passed through a three-stage filtering process: (A) Style Debiasing, (B) Contrastive Debiasing, and (C) Alternative Projection via ICA, prior to the final UMAP visualization. Compared to the unprocessed projection (1), the processed version (2) exhibits improved class separability and reduced influence of style artifacts, aiding visual verification of class membership.
• Figure 3: Explanation pipeline for visual and textual attribution in CIR. The system accepts a reference image $I_r$ and a relative caption $C$, combining them into the prompt "a photo of $\varphi(I_r)$ that $C$", where $\varphi(I_r)$ is a pseudo-token generated via a textual inversion network. This prompt is tokenized and encoded by CLIP's text encoder to produce a text embedding $T$. Candidate images from the dataset (e.g., ImageNet-R) are encoded into image embeddings $V$, and retrieval is based on cosine similarity $s = \cos(V, T)$. To generate explanations, the similarity score $s$ is differentiated with respect to either the image or the text tokens. The resulting gradients yield visual saliency maps (top right) and token attribution scores (bottom right), helping users understand which parts of the image or prompt influenced the retrieval result most.
• Figure 4: Reference and target image pairs used in the quantitative evaluation. Task A (porcupine) and Task B (Boston Terrier) each required retrieving a stylized version of a realistic reference image. Participants iteratively refined a text prompt until the target appeared within the top-3 retrieved results.