XR: Cross-Modal Agents for Composed Image Retrieval

TL;DR

XR introduces a training-free cross-modal multi-agent framework for composed image retrieval that orchestrates imagination, coarse filtering, and fine filtering to align results with user edits. It leverages cross-modal captions and verification questions to progressively refine a candidate set, combining similarity-based ranking with factual checks via a weighted fusion scheme. Across CIRR, CIRCO, and FashionIQ, XR achieves consistent, significant gains over both zero-shot and supervised baselines, with ablations confirming the essential role of each module. This approach demonstrates the value of retrieval-informed, cross-modal reasoning for robust, user-aligned multimodal search and points to scalable extensions to richer modalities and interactive queries.

Abstract

Retrieval is being redefined by agentic AI, demanding multimodal reasoning beyond conventional similarity-based paradigms. Composed Image Retrieval (CIR) exemplifies this shift as each query combines a reference image with textual modifications, requiring compositional understanding across modalities. While embedding-based CIR methods have achieved progress, they remain narrow in perspective, capturing limited cross-modal cues and lacking semantic reasoning. To address these limitations, we introduce XR, a training-free multi-agent framework that reframes retrieval as a progressively coordinated reasoning process. It orchestrates three specialized types of agents: imagination agents synthesize target representations through cross-modal generation, similarity agents perform coarse filtering via hybrid matching, and question agents verify factual consistency through targeted reasoning for fine filtering. Through progressive multi-agent coordination, XR iteratively refines retrieval to meet both semantic and visual query constraints, achieving up to a 38% gain over strong training-free and training-based baselines on FashionIQ, CIRR, and CIRCO, while ablations show each agent is essential. Code is available: https://01yzzyu.github.io/xr.github.io/.

Figures (7)

• Figure 1: Framework of $\text{X}^\text{R}$. The multi-agent system integrates textual and visual imagination with cross-modal similarity and question-based scoring, followed by re-ranking. This multi-stage reasoning process exploits complementary cues from both modalities, effectively handling fine-grained modifications that single-modality approaches often miss.
• Figure 2: Parameter analysis of $\text{X}^\text{R}$. (a) Effect of RRF with different $z$ values. (b) Comparison across multimodal backbones. (c) Impact of the number of verification questions. (d) Influence of candidate pool size $k^\prime$.
• Figure 3: Effect of $\lambda$ on text–image fusion: best at $\lambda{=}0.15$; extremes degrade by losing cross-modal cues.
• Figure 4: Latency of $\text{X}^\text{R}$ under different top-$k^\prime$: larger pools increase cost nearly linearly, but $k^\prime \approx 100$ balances coverage and overhead.
• Figure F.1: Case study on CIRR. $\text{X}^\text{R}$ correctly grounds complex scene edits (e.g., bus orientation, reflective jackets) through factual verification. Target image is marked with the green box.