Scale Up Composed Image Retrieval Learning via Modification Text Generation

TL;DR

This work tackles data scarcity in Composed Image Retrieval by training a Modification Text-oriented Synthetic Triplets (MTST) generator that converts image pairs into expressive modification texts. It leverages MTST for large-scale pretraining and introduces Prototypical Two-Hop Alignment (PTHA), which first learns an implicit prototype via reverse text and then fuses it with the modifier to align with the target image. Across natural and fashion domains, MTST pretraining plus PTHA yields competitive CIRR and FashionIQ performance, with notable zero-shot generalization to CIRCOcirco and strong data-efficiency compared with other synthetic-data approaches. The approach demonstrates the value of synthetic, high-quality modification text for scalable, cross-modal retrieval without extensive manual annotation, enabling faster domain adaptation and improved downstream CIR accuracy.

Abstract

Composed Image Retrieval (CIR) aims to search an image of interest using a combination of a reference image and modification text as the query. Despite recent advancements, this task remains challenging due to limited training data and laborious triplet annotation processes. To address this issue, this paper proposes to synthesize the training triplets to augment the training resource for the CIR problem. Specifically, we commence by training a modification text generator exploiting large-scale multimodal models and scale up the CIR learning throughout both the pretraining and fine-tuning stages. During pretraining, we leverage the trained generator to directly create Modification Text-oriented Synthetic Triplets(MTST) conditioned on pairs of images. For fine-tuning, we first synthesize reverse modification text to connect the target image back to the reference image. Subsequently, we devise a two-hop alignment strategy to incrementally close the semantic gap between the multimodal pair and the target image. We initially learn an implicit prototype utilizing both the original triplet and its reversed version in a cycle manner, followed by combining the implicit prototype feature with the modification text to facilitate accurate alignment with the target image. Extensive experiments validate the efficacy of the generated triplets and confirm that our proposed methodology attains competitive recall on both the CIRR and FashionIQ benchmarks.

Figures (8)

• Figure 1: Comparison of existing automatic modifier generation method and ours. Above: Existing methods utilize indirect information, such as labels and captions of images to generate modifiers, which often results in poorer notation quality, limited text length, and a lack of diversity in textual form. Below: Our proposed method combines the image pair and instruction, mapping them into a frozen LLM for text generation. This allows for a more flexible description of the details between images and generates high-quality modification text with controllable length.
• Figure 2: An intuitive prototype example of the reference image's prototype reached by the modifier and the target image's prototype reached by the reversed modifier. The green dashed line represents the content preserved in the image based on the corresponding modification text. The blue dashed line represents the modifications and additions made to the prototype based on the corresponding modified text.
• Figure 3: An overview of MTST Generator architecture. The paired reference images and target images are fed into an Image Encoder and a trainable Query Encoder to get their respective features ${q_r,q_t}$. These representations are then concatenated and fused with the instruction using the same Query Encoder to obtain a fusion representation ${q_c}$. Representations ${q_r,q_t,q_c}$ along with the instruction embedding are concatenated and fed into a frozen LLM to generate modification text.
• Figure 4: Top: Overview of the training phase. We first employ implicit prototype learning between ${f_{r2t}}$ and detached ${f_{t2r}}$. ${f_{t2r}}$ is extracted from the target image and generated reversed text using the MTST generator. In the second step, we utilize contrastive loss between fusion feature $f_q$ and target image ${f_t}$, text-only feature ${f_m}$, and target image feature ${f_t}$. Bottom: Overview of the inference phase. We leverage the fusion feature $f_q$ to compute similarity with the features extracted from the image gallery to perform retrieval.
• Figure 5: Left: Ablation studies on different pre-training ${\text{CIRR}_\text{MTST}}$ data size. We report Recall@5 and Avg. metric on CIRR validation set by fine-tuning with \ref{['finaloss']}. Right: We deploy different ${L_{p2p}}$ weight ${\alpha}$ on fine-tuning stage with the same pre-trained model.