Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation

TL;DR

The paper tackles Composed Image Retrieval (CIR) data scarcity by introducing automatic triplet synthesis via counterfactual image generation. It builds training triplets $\langle I_{\rm{ref}}, t, I_{\rm{target}} \rangle$ through a two-stage process: generating a counterfactual caption $c_{\rm{cf}}$ from a reference caption $c_{\rm{ref}}$ using LANCE, and synthesizing $I_{\rm{target}}$ with Stable Diffusion guided by prompt-to-prompt editing and null-text inversion to preserve fidelity to $I_{\rm{ref}}$ while applying localized changes. The main contributions are (1) a fully automatic triplet synthesis pipeline for CIR and (2) empirical evidence that synthetic triplets improve CIR performance in data-scarce regimes on CIRR and FashionIQ. This approach enables scalable generation of high-quality, diverse CIR training data and enhances retrieval accuracy for user-specified modifications.

Abstract

Composed Image Retrieval (CIR) provides an effective way to manage and access large-scale visual data. Construction of the CIR model utilizes triplets that consist of a reference image, modification text describing desired changes, and a target image that reflects these changes. For effectively training CIR models, extensive manual annotation to construct high-quality training datasets, which can be time-consuming and labor-intensive, is required. To deal with this problem, this paper proposes a novel triplet synthesis method by leveraging counterfactual image generation. By controlling visual feature modifications via counterfactual image generation, our approach automatically generates diverse training triplets without any manual intervention. This approach facilitates the creation of larger and more expressive datasets, leading to the improvement of CIR model's performance.

Figures (4)

• Figure 1: Overview of the proposed triplet synthesis method for enhancing CIR performance. This figure shows the process of synthesizing triplets. It involves modifying the reference caption $c_{\rm{ref}}$ to create a counterfactual caption $c_{\rm{cf}}$ and generating its corresponding image.
• Figure 2: Retrieved results on the CIRR and FashionIQ datasets. It highlights that the proposed synthetic triplet method enables the acquisition of essential characteristics of images. Note that the ground truth image is indicated by a green box.
• Figure 3: Examples of triplets synthesized by the proposed method. This figure shows an example of target image generation modified by the modification text shown in red. The global structure remains intact, while only local changes are reflected in the synthesized images.
• Figure 4: Comparison of R@k across different training image proportions and synthetic triplets. This figure shows R@1, R@5, R@10, and R@50 for different numbers of training images, with and without the addition of 5,000 synthetic triplets.