SpatialFormer: Semantic and Target Aware Attentions for Few-Shot Learning

TL;DR

This work tackles two core challenges in few-shot learning: inaccurate attention maps from CNN-based cross-attention and distraction from background contexts. It introduces SpatialFormer, a Transformer-based module that emphasizes semantic-level similarity using a reference object, and combines it with Semantic and Target Attentions (STA) comprising SFSA and SFTA within the STANet framework. A Novel Task Attention (NTA) and a multi-task loss are added to boost inter-class separability and task adaptation, yielding state-of-the-art results on miniImageNet, tieredImageNet, and CIFAR-FS. Extensive ablations and visualizations corroborate the effectiveness of focusing attention on target objects and leveraging base-class semantic information for few-shot classification.

Abstract

Recent Few-Shot Learning (FSL) methods put emphasis on generating a discriminative embedding features to precisely measure the similarity between support and query sets. Current CNN-based cross-attention approaches generate discriminative representations via enhancing the mutually semantic similar regions of support and query pairs. However, it suffers from two problems: CNN structure produces inaccurate attention map based on local features, and mutually similar backgrounds cause distraction. To alleviate these problems, we design a novel SpatialFormer structure to generate more accurate attention regions based on global features. Different from the traditional Transformer modeling intrinsic instance-level similarity which causes accuracy degradation in FSL, our SpatialFormer explores the semantic-level similarity between pair inputs to boost the performance. Then we derive two specific attention modules, named SpatialFormer Semantic Attention (SFSA) and SpatialFormer Target Attention (SFTA), to enhance the target object regions while reduce the background distraction. Particularly, SFSA highlights the regions with same semantic information between pair features, and SFTA finds potential foreground object regions of novel feature that are similar to base categories. Extensive experiments show that our methods are effective and achieve new state-of-the-art results on few-shot classification benchmarks.

Figures (7)

• Figure 1: As shown in 1st column, the limitation of cross-attention based approaches is that they only recognize the mutual similar object, but can not solve the distraction problem causing by similar backgrounds. In this paper, we propose two effective modules called SFSA and SFTA, which are utilized to find the mutual similar object and obtain target-aware information, respectively. With their combination, the derived STA module is able to enhance target object while alleviate background distraction.
• Figure 2: (a) shows a standard Transformer structure. (b) is the proposed SpatialFormer, where ${(f,r)}$ are input feature and reference object, and SpatialAttention operation is presented in (c). (d) gives the intuition for SpatialFormer, which uses ${A}$ (i.e. ${Q}$-aligned ${V}$) to strength the spatial response of ${Q}$. And the dimension of each node (i.e. the circle in (d)) is $\mathbb{R}^{c}$, which denotes each spatial position of features. The gray arrow represents low similarity relationship, and the colorful arrow is high similarity.
• Figure 3: The STANet framework. (a) The designed Novel Task Attention (NTA) module obtains novel task information via utilizing the weights of Novel Classifier, which is fine-tuned on support data of novel set ${X^{novel}}$. (b) The proposed Semantic and Target Attentions (STA) module enhances target object regions. Specifically, SpatialFormer Semantic Attention (SFSA) module highlights mutually similar regions between support and query features. Meanwhile, SpatialFormer Target Attention (SFTA) module highlights potential foreground object regions via utilizing base target information embedded in Global Classifier.
• Figure 4: The designed Novel Task Attention (NTA) module obtains novel task information via utilizing the weights of Novel Classifier.
• Figure 5: The class activation maps generated from 5-way 5-shot task, where Embedding means the features extracted after the backbone in STANet.