Enhancing Environmental Robustness in Few-shot Learning via Conditional Representation Learning

TL;DR

This work addresses the gap between laboratory-FSL performance and real-world robustness by introducing the RD-FSL real-world multi-domain benchmark and a conditional representation learning network (CRLNet). CRLNet jointly processes support and query images through a feature extractor, a conditional learner, and a re-representation learner, guided by cross-attention and 4D convolution to generate more discriminative representations, reinforced by a contrastive loss. The paper demonstrates that CRLNet outperforms state-of-the-art methods by 6.83%–16.98% across diverse datasets and backbones, verifies its robustness through ablations and visual analyses, and releases code and data for reproducibility. The proposed RD-FSL benchmark and CRLNet offer tangible improvements for practical few-shot recognition in complex environments, with potential impact on domain-specific visual recognition tasks such as biology, mining, archaeology, and agriculture.

Abstract

Few-shot learning (FSL) has recently been extensively utilized to overcome the scarcity of training data in domain-specific visual recognition. In real-world scenarios, environmental factors such as complex backgrounds, varying lighting conditions, long-distance shooting, and moving targets often cause test images to exhibit numerous incomplete targets or noise disruptions. However, current research on evaluation datasets and methodologies has largely ignored the concept of "environmental robustness", which refers to maintaining consistent performance in complex and diverse physical environments. This neglect has led to a notable decline in the performance of FSL models during practical testing compared to their training performance. To bridge this gap, we introduce a new real-world multi-domain few-shot learning (RD-FSL) benchmark, which includes four domains and six evaluation datasets. The test images in this benchmark feature various challenging elements, such as camouflaged objects, small targets, and blurriness. Our evaluation experiments reveal that existing methods struggle to utilize training images effectively to generate accurate feature representations for challenging test images. To address this problem, we propose a novel conditional representation learning network (CRLNet) that integrates the interactions between training and testing images as conditional information in their respective representation processes. The main goal is to reduce intra-class variance or enhance inter-class variance at the feature representation level. Finally, comparative experiments reveal that CRLNet surpasses the current state-of-the-art methods, achieving performance improvements ranging from 6.83% to 16.98% across diverse settings and backbones. The source code and dataset are available at https://github.com/guoqianyu-alberta/Conditional-Representation-Learning.

Figures (8)

• Figure 1: Motivation for enhancing "environmental robustness" in few-shot learning. Even within the same category, like "butterflies", real-world data presents numerous complexities, including camouflaged targets, incomplete targets, and image blurriness, as compared to training data. When features are extracted from both training and real-world data using classical feature extractors like ResNet-50 HeZRS16 and ViT abs-2010-11929, there is a significant discrepancy in feature distributions for the same category.
• Figure 2: Comparison between the framework of the (I) baseline and (II) the proposed conditional representation learning. In the proposed conditional representation learning framework, the conditional learner and re-representation learner further optimize the prototype features output by the feature extractor. This enhancement improves the expression of information related to class-discriminative features.
• Figure 3: (I) The construction process of the real-world multi-domain few-shot visual recognition (RD-FSL) benchmark includes three steps: data collection, data cleaning, and manual annotation. The manual annotation step involves human labeling of fine-grained classification labels and difficulty (support/query) labels. Additionally, (II) examples of images and annotations within it demonstrate the contrast in difficulty levels for the same category labeled as support and query across six validation datasets.
• Figure 4: The overview of the proposed conditional representation learning network (CRLNet) includes a feature extractor, a conditional learner, and a re-representation learner. The feature extractor maps support $\mathcal{I}^{s}$ and query images $\mathcal{I}^{q}$ to prototype feature matrices ${f}^{s}$ and ${q}^{s}$. Subsequently, the conditional learner learns the conditional matrices $\omega^{s}_{c}$ and $\omega^{q}_{c}$. Finally, the re-representation learner relearns the ${f}^{s}$ and ${q}^{s}$ with $\omega^{s}_{c}$ and $\omega^{q}_{c}$ to obtain the final feature representation $\mathcal{F}^{s}$ and $\mathcal{F}^{q}$. The entire CRLNet is supervised with a contrastive learning loss.
• Figure 5: The comparison experiment results between CRLNet and the baseline FinnAL17 on the RD-FSL Benchmark with ResNet-12 show significant improvements. The numbers above the arrows indicate the performance enhancement of CRLNet compared to the baseline.