CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective

TL;DR

<3-5 sentence high-level summary> This work tackles FS-FGVC by reframing it as a causal inference problem under a structural causal model, where the few-shot selection and intrinsic fine-grained variability act as an unobservable confounder. It introduces two mechanisms, the Interventional Multi-Scale Encoder (IMSE) and Interventional Masked Feature Reconstruction (IMFR), to perform sample-level and feature-level interventions via frontdoor adjustment, yielding $P(Y|do(X))$ that reflects true causality from inputs to subcategories. Extensive experiments on CUB-200-2011, Stanford Dogs, and Stanford Cars demonstrate state-of-the-art performance, with ablations confirming the complementary benefits of IMSE and IMFR. The approach provides a principled, causally grounded path to robust FS-FGVC under biased sampling, with potential for integration into unified architectures.

Abstract

Few-shot fine-grained visual categorization (FS-FGVC) focuses on identifying various subcategories within a common superclass given just one or few support examples. Most existing methods aim to boost classification accuracy by enriching the extracted features with discriminative part-level details. However, they often overlook the fact that the set of support samples acts as a confounding variable, which hampers the FS-FGVC performance by introducing biased data distribution and misguiding the extraction of discriminative features. To address this issue, we propose a new causal FS-FGVC (CausalFSFG) approach inspired by causal inference for addressing biased data distributions through causal intervention. Specifically, based on the structural causal model (SCM), we argue that FS-FGVC infers the subcategories (i.e., effect) from the inputs (i.e., cause), whereas both the few-shot condition disturbance and the inherent fine-grained nature (i.e., large intra-class variance and small inter-class variance) lead to unobservable variables that bring spurious correlations, compromising the final classification performance. To further eliminate the spurious correlations, our CausalFSFG approach incorporates two key components: (1) Interventional multi-scale encoder (IMSE) conducts sample-level interventions, (2) Interventional masked feature reconstruction (IMFR) conducts feature-level interventions, which together reveal real causalities from inputs to subcategories. Extensive experiments and thorough analyses on the widely-used public datasets, including CUB-200-2011, Stanford Dogs, and Stanford Cars, demonstrate that our CausalFSFG achieves new state-of-the-art performance. The code is available at https://github.com/PKU-ICST-MIPL/CausalFSFG_TMM.

Figures (5)

• Figure 1: (a) Illustration of the distribution bias caused by the selection operation under the few-shot condition. (b) Illustration of aligning the FS-FGVC problem with the Structural Causal Model assumption, aiming to infer the accurate prediction (effect) from the input (cause) through extracted features (mediator). (c) Illustration of the inherent fine-grained nature of the fine-grained data.
• Figure 2: The Structural Causal Model assumptions for the FS-FGVC problem, dashed lines mean the causality is unobservable during training. (a) In the FGVC context, the confounder is observable since the observable data coincides with the full dataset. (b) In the FS-FGVC context, the biased distribution of the selected observable data makes the confounder unobservable.
• Figure 3: The overall framework of our proposed CausalFSFG approach. The Interventional Multi-Scale Encoder module implements the sample-level intervention to extract the intervened features. Then, the Interventional Masked Feature Reconstruction module further implements the feature-level intervention to improve the classification performance.
• Figure 4: The visualization examples of the proposed CausalFSFG framework on the CUB dataset with the ResNet-12 backbone. The top block presents the input samples. The middle block presents the visualizations of the multi-scale features extracted by the first to fourth layers of the backbone, respectively. The bottom block presents the visualizations of the intervened feature generated by our IMSE module.
• Figure 5: Visualizations of erroneous results on the CUB test set. Red boxes indicate false image contents identified by the model, and green boxes highlight discriminative image contents for correct classification.