ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval

TL;DR

This work identifies Capability Degradation when adapting generative Multimodal Large Language Models to discriminative CIR retrieval, where fine-grained compositional reasoning is lost. It proposes ReCALL, a model-agnostic diagnose-generate-refine framework that leverages a foundation model’s native reasoning to recalibrate the retriever’s embedding space through self-guided mining, CoT-based corrective generation, VQA-based filtering, and grouped contrastive refinement. The method demonstrates state-of-the-art results on CIRR and FashionIQ, validating its effectiveness in recovering fine-grained visual-semantic distinctions and enhancing robustness against hard negatives. The approach offers a scalable path for integrating powerful MLLMs into retrieval systems without sacrificing their core reasoning capabilities, with potential broad impact on CIR applications and beyond.

Abstract

Composed Image Retrieval (CIR) aims to retrieve target images based on a hybrid query comprising a reference image and a modification text. Early dual-tower Vision-Language Models (VLMs) struggle with cross-modality compositional reasoning required for this task. Recently, adapting generative Multimodal Large Language Models (MLLMs) for retrieval offers a promising direction. However, we identify that this adaptation strategy overlooks a fundamental issue: adapting a generative MLLM into a single-embedding discriminative retriever triggers a paradigm conflict, which leads to Capability Degradation - the deterioration of native fine-grained reasoning after retrieval adaptation. To address this challenge, we propose ReCALL (Recalibrating Capability Degradation), a model-agnostic framework that follows a diagnose-generate-refine pipeline: Firstly, we diagnose cognitive blind spots of the retriever via self-guided informative instance mining. Next, we generate corrective instructions and triplets by CoT prompting the foundation MLLM and conduct quality control with VQA-based consistency filtering. Finally, we refine the retriever through continual training on these triplets with a grouped contrastive scheme, thereby internalizing fine-grained visual-semantic distinctions and realigning the discriminative embedding space of retriever with intrinsic compositional reasoning within the MLLM. Extensive experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance. Code will be released soon.

Figures (4)

• Figure 1: Empirical illustration of Capability Degradation and the effectiveness of ReCALL ($\mathcal{R}_{\text{refine}}$).(a) We compare the Foundation MLLM ($\mathcal{F}$) under its native VQA-based generative paradigm with its fine-tuned retrieval counterpart ($\mathcal{R}_{\text{base}}$) under a similarity-based discriminative paradigm using a challenging query that requires fine-grained reasoning. The base retriever $\mathcal{R}_{\text{base}}$ fails due to fine-grained grounding errors, while $\mathcal{F}$ succeeds through step-wise reasoning.(b) Quantitative evidence of Capability Degradation and Recalibrate. We test $\mathcal{R}_{\text{base}}$ on a subset of 1k instances where $\mathcal{F}$ successfully retrieves the target (i.e., $\mathcal{F}$ achieves 100% R@1). The low R@1 performance of $\mathcal{R}_{\text{base}}$ (only 62.33% on CIRR and 55.80% on FashionIQ) on this $\mathcal{F}$-solvable subset provides quantifiable proof of capability degradation. Our proposed ReCALL framework effectively recovers the lost abilities, boosting the performance of $\mathcal{R}_{\text{base}}$ to $\mathcal{R}_{\text{refine}}$, with significant gains.
• Figure 2: Overview of the ReCALL framework. (1) Stage 1: A baseline retriever $\mathcal{R}_{base}$ is adapted from the foundation model $\mathcal{F}$ via standard fine-tuning. (2) Stage 2 (Diagnose):$\mathcal{R}_{base}$ surfaces its own failure cases via self-guided informative instance mining. (3) Stage 3 (Generate): Leveraging native reasoning (CoT), $\mathcal{F}$ synthesizes minimally edited corrective instructions for the mined informative instances. (4) Stage 4 (Refine): Based on the original and enhanced triplets, a Grouped Contrastive Refinement strategy is employed to produce the final $\mathcal{R}_{refine}$, effectively recalibrating the degraded capabilities.
• Figure 3: Qualitative comparison between the baseline ($\mathcal{R}_{\text{base}}$) and our ReCALL ($\mathcal{R}_{\text{refine}}$) on FashionIQ (top) and CIRR (bottom). The green dashed boxes indicate the ground-truth targets. $\mathcal{R}_{\text{base}}$ suffers from capability degradation, failing to capture specific details like "half sleeved" or "facing the camera," while ReCALL successfully retrieves the correct targets by recalibrating these fine-grained reasoning.
• Figure 4: Generalizability across backbones. We validate ReCALL on stronger foundation models (Qwen2.5-VL-7B and Qwen3-VL-8B). Despite higher baselines, ReCALL consistently delivers performance gains on both (a) CIRR and (b) FashionIQ, confirming the strong generalization of our framework.