DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines

TL;DR

The paper tackles FGIR by formulating practical guidelines (G1–G3) to design discriminative and generalizable embeddings and introduces DVF, a ViT-based framework with an Object-oriented Visual Filtering module using Grounding-DINO for training-free object magnification and a Semantic-oriented Visual Filtering module guided by a Token Importance Generator $\Omega$ to select discriminative tokens. It also presents a Discriminative Model Training (DMT) strategy that combines data augmentation with a contrastive loss $L_{con}$, alongside the ProxyNCA loss $L_{pnca}$, establishing the objective $L = L_{pnca} + L_{con}$. Through evaluations on CUB-200-2011, Stanford Cars 196, and NABirds, the method achieves state-of-the-art performance in both closed-set and open-set FGIR and is supported by visualizations that illustrate precise, discriminative region emphasis. The work demonstrates the practical impact of guideline-driven FGIR design and offers a training-free object localization component (OVF) and token-focused refinement (SVF) that together deliver robust, interpretable FGIR performance.$

Abstract

Fine-grained image retrieval (FGIR) is to learn visual representations that distinguish visually similar objects while maintaining generalization. Existing methods propose to generate discriminative features, but rarely consider the particularity of the FGIR task itself. This paper presents a meticulous analysis leading to the proposal of practical guidelines to identify subcategory-specific discrepancies and generate discriminative features to design effective FGIR models. These guidelines include emphasizing the object (G1), highlighting subcategory-specific discrepancies (G2), and employing effective training strategy (G3). Following G1 and G2, we design a novel Dual Visual Filtering mechanism for the plain visual transformer, denoted as DVF, to capture subcategory-specific discrepancies. Specifically, the dual visual filtering mechanism comprises an object-oriented module and a semantic-oriented module. These components serve to magnify objects and identify discriminative regions, respectively. Following G3, we implement a discriminative model training strategy to improve the discriminability and generalization ability of DVF. Extensive analysis and ablation studies confirm the efficacy of our proposed guidelines. Without bells and whistles, the proposed DVF achieves state-of-the-art performance on three widely-used fine-grained datasets in closed-set and open-set settings.

Figures (7)

• Figure 1: We compare our method, which adheres to the guidelines, with the state-of-the-art Hyp-ViT, which violates some of these guidelines. Hyp-ViT makes retrieval errors in cases of similar images due to the objects being too small to extract subcategory-specific discrepancies. In contrast, our DVF succeeds by enlarging the objects.
• Figure 2: Overview of the proposed framework. The framework consists of two core modules, 1) Object-oriented Visual Filtering Module: utilizes a visual foundation model to zoom in object in the input image (details are in Section \ref{['oovf']}); 2) Semantic-oriented Visual Filtering Module: accounts for the attention of the class token as well as the importance of the embedding token itself to locate discriminative regions in the object (details are in Section \ref{['sovf']}).
• Figure 3: (a),(d) origin input image, (b),(e) object-oriented visual filtering without post-processing, (c),(f) object-oriented visual filtering with post-processing. (g) the evaluation results (%) on CUB-200-2011 with/without post-processing.
• Figure 4: Analyses of hyper-parameter $k$ on CUB-200-2011.
• Figure 5: Visualization of SVF with/without token importance on CUB-200-2011. The token importance empowers DVF to concentrate on discriminative regions, including the beak, tail, and areas with color mutations.