FIGROTD: A Friendly-to-Handle Dataset for Image Guided Retrieval with Optional Text

TL;DR

The paper addresses the fragmentation and resource demands of Image-Guided Retrieval with Optional Text (IGROT) by introducing FIGROTD, a lightweight yet high-quality benchmark spanning CIR, SBIR, and CSTBIR. It presents VaGFeM, a variance-guided feature mask that fuses image and text into a discriminative shared embedding, along with a dual-loss objective that combines InfoNCE and Triplet losses to promote compositional reasoning while preserving cross-modal alignment. Empirically, VaGFeM achieves competitive results across nine benchmarks with far fewer triplets than large-scale alternatives, and is shown to be data-efficient with a peak performance around 10K triplets. The work highlights a balanced, data-efficient pathway for IGROT and sheds light on the trade-offs of incorporating triplet supervision across diverse retrieval subtasks.

Abstract

Image-Guided Retrieval with Optional Text (IGROT) unifies visual retrieval (without text) and composed retrieval (with text). Despite its relevance in applications like Google Image and Bing, progress has been limited by the lack of an accessible benchmark and methods that balance performance across subtasks. Large-scale datasets such as MagicLens are comprehensive but computationally prohibitive, while existing models often favor either visual or compositional queries. We introduce FIGROTD, a lightweight yet high-quality IGROT dataset with 16,474 training triplets and 1,262 test triplets across CIR, SBIR, and CSTBIR. To reduce redundancy, we propose the Variance Guided Feature Mask (VaGFeM), which selectively enhances discriminative dimensions based on variance statistics. We further adopt a dual-loss design (InfoNCE + Triplet) to improve compositional reasoning. Trained on FIGROTD, VaGFeM achieves competitive results on nine benchmarks, reaching 34.8 mAP@10 on CIRCO and 75.7 mAP@200 on Sketchy, outperforming stronger baselines despite fewer triplets.

Figures (4)

• Figure 1: Illustration of our proposed methodology. (a) VaGFeM enhances query features by applying variance-guided masking on fused embeddings. (b) Training employs both InfoNCE loss and triplet loss to improve compositional discrimination in a shared embedding space.
• Figure 2: Comparison of cosine similarity distributions, highlighting the superior alignment of our proposed VaGFeM method (left) in Figure \ref{['fig:vagfem_sim']} which produces a single, high-similarity peak, against the bimodal distribution (Figure \ref{['fig:transagg_sim']}) of the baseline TransAgg liu2023zeroshot model (right).
• Figure 3: Comparison of model performance across different loss functions and training data sizes on nine benchmarks. (a) Impact of loss type (infonce vs. infonce+triplet). (b) Effect of varying training data size (1K, 2K, 5K, 10K, all).
• Figure 4: Qualitative retrieval examples on the three FIGROTD tasks. VaGFeM achieves strong results on CIR and SBIR, but shows limited effectiveness on CSTBIR, where fine-grained compositional reasoning is required.