CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval

TL;DR

CSMCIR addresses representation space fragmentation in Composed Image Retrieval by generating discriminative target captions with Multi-level Chain-of-Thought prompting, then enforcing modal symmetry with a shared-parameter Q-Former across query and target branches. A symmetric dual-tower architecture processes $(I_r,T)$ and $(I_t,T(I_t))$ with identical encoders, while an entropy-aware Memory Bank provides high-quality, temporally consistent negatives for efficient contrastive learning. The approach yields state-of-the-art results on Fashion-IQ, CIRR, Shoes, and LaSCO, with notable training and inference efficiency gains and robust ablation-supported contributions. This framework reduces reliance on post-hoc alignment, enabling more reliable and scalable CIR in real-world applications like e-commerce search and interactive visual querying.

Abstract

Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text, offering substantial advantages over single-modality retrieval systems. However, existing CIR methods suffer from representation space fragmentation: queries and targets comprise heterogeneous modalities and are processed by distinct encoders, forcing models to bridge misaligned representation spaces only through post-hoc alignment, which fundamentally limits retrieval performance. This architectural asymmetry manifests as three distinct, well-separated clusters in the feature space, directly demonstrating how heterogeneous modalities create fundamentally misaligned representation spaces from initialization. In this work, we propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components. First, we introduce a Multi-level Chain-of-Thought (MCoT) prompting strategy that guides Multimodal Large Language Models to generate discriminative, semantically compatible captions for target images, establishing modal symmetry. Building upon this, we design a symmetric dual-tower architecture where both query and target sides utilize the identical shared-parameter Q-Former for cross-modal encoding, ensuring consistent feature representations and further reducing the alignment gap. Finally, this architectural symmetry enables an entropy-based, temporally dynamic Memory Bank strategy that provides high-quality negative samples while maintaining consistency with the evolving model state. Extensive experiments on four benchmark datasets demonstrate that our CSMCIR achieves state-of-the-art performance with superior training efficiency. Comprehensive ablation studies further validate the effectiveness of each proposed component.

Figures (11)

• Figure 1: Workflows of existing CIR methods (a) and our proposed CSMCIR (b). Our approach achieves modal and structural symmetry for better alignment.
• Figure 2: T-SNE visualization comparing SPRC and CSMCIR on Fashion-IQ dresses.
• Figure 3: Overview of CSMCIR framework. Right: MCoT-based target caption generation. Left: (a) Entropy-Aware Memory Bank for negative sampling. (b) Memory Bank update via entropy-based and temporal scoring.
• Figure 4: Qualitative comparison with SPRC. Green outlines indicate successfully retrieved targets.
• Figure 5: Ablation study on Memory Bank strategies.