Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition

TL;DR

RSaD tackles FS-FGVR by integrating saliency priors into a mutual-learning framework that distills discriminative object regions into a compact embedding. It combines Saliency-aware Guidance (SaG) to align saliency distributions and Representation Highlight&Summarize (RHS) to learn a transferable contextual representation via prototype-relational highlighting and summarization. Empirical results on CUB-200-2011, Stanford Dogs, and Stanford Cars show competitive accuracy with lower computational overhead than many baselines, validating the efficiency and effectiveness of saliency-guided, low-dimensional supervision for few-shot fine-grained tasks. The approach demonstrates that incorporating saliency priors in a bidirectional distillation framework can significantly improve generalization to unseen sub-categories in data-scarce regimes.

Abstract

Recognizing novel sub-categories with scarce samples is an essential and challenging research topic in computer vision. Existing literature addresses this challenge by employing local-based representation approaches, which may not sufficiently facilitate meaningful object-specific semantic understanding, leading to a reliance on apparent background correlations. Moreover, they primarily rely on high-dimensional local descriptors to construct complex embedding space, potentially limiting the generalization. To address the above challenges, this article proposes a novel model, Robust Saliency-aware Distillation (RSaD), for few-shot fine-grained visual recognition. RSaD introduces additional saliency-aware supervision via saliency detection to guide the model toward focusing on the intrinsic discriminative regions. Specifically, RSaD utilizes the saliency detection model to emphasize the critical regions of each sub-category, providing additional object-specific information for fine-grained prediction. RSaD transfers such information with two symmetric branches in a mutual learning paradigm. Furthermore, RSaD exploits inter-regional relationships to enhance the informativeness of the representation and subsequently summarize the highlighted details into contextual embeddings to facilitate the effective transfer, enabling quick generalization to novel sub-categories. The proposed approach is empirically evaluated on three widely used benchmarks, demonstrating its superior performance.

Figures (6)

• Figure 1: The statistical analysis of both external environments and intrinsic attributes across two sub-categories within the CUB-200-2011 dataset. The results reveal that 1) environments and specific attributes across different sub-categories may exhibit significant similarities; 2) within the same sub-category, identical attributes can vary considerably.
• Figure 2: The framework of the proposed RSaD for few-shot fine-grained visual recognition. This framework consists of three hierarchical levels of operations. At the image level, the framework generates the saliency serving as input for the symmetric branch. At the feature level, it highlights crucial features while aggregating significant information. Finally, the two branches independently optimize the cross-entropy (CE) loss while simultaneously providing complementary signals at the distribution level via mutual learning.
• Figure 3: Augmented Saliency Generation. For the input image, this module generates multiple saliency maps. Next, binarization operations are performed on these maps, followed by an OR operation. Then, the priors are synthesized through the Hadamard product of the input image and mask.
• Figure 4: Visual comparison of the ensemble strategy with the individual model. The saliency produced by the ensemble model is more precise and more reliable than others.
• Figure 5: The visualization of local regions obtained from Grad-CAM. (a) prototype, (b) baseline, (c) our RSaD w/o RHS, and (d) query. The Grad-CAM Map identifies significant regions in the input image that affect classification decisions, where darker regions indicate higher importance. The specimens within each column belong to the same subclass, while those in different columns belong to distinct classes. Compared with baseline, RSaD pays more attention to the distinguishing characteristics of objects themselves.