LPN: Language-guided Prototypical Network for few-shot classification

TL;DR

A Language-guided Prototypical Network for few-shot classification without image-level captions is proposed, which outperforms several state-of-the-art methods on benchmark datasets, showcasing its effectiveness and robustness.

Abstract

Few-shot classification aims to adapt to new tasks with limited labeled examples. To fully use the accessible data, recent methods explore suitable measures for the similarity between the query and support images and better high-dimensional features with meta-training and pre-training strategies. However, the potential of multi-modality information has barely been explored, which may bring promising improvement for few-shot classification. In this paper, we propose a Language-guided Prototypical Network (LPN) for few-shot classification, which leverages the complementarity of vision and language modalities via two parallel branches to improve the classifier. Concretely, to introduce language modality with limited samples in the visual task, we leverage a pre-trained text encoder to extract class-level text features directly from class names while processing images with a conventional image encoder. Then, we introduce a language-guided decoder to obtain text features corresponding to each image by aligning class-level features with visual features. Additionally, we utilize class-level features and prototypes to build a refined prototypical head, which generates robust prototypes in the text branch for follow-up measurement. Furthermore, we leverage the class-level features to align the visual features, capturing more class-relevant visual features. Finally, we aggregate the visual and text logits to calibrate the deviation of a single modality, enhancing the overall performance. Extensive experiments demonstrate the competitiveness of LPN against state-of-the-art methods on benchmark datasets.

Figures (6)

• Figure 1: Illustration of LPN for three-way two-shot few-shot classification. Given the task, we generate feature maps $f_v$ with the image encoder and build visual logits $V_s$ through a conventional metric module. As for the language modality, we leverage the pre-trained text encoder to extract class-level text features $f_t^{cl}$. Then, we propose a language-guided decoder to obtain the corresponding text features. LPN computes the text logits $T_s$ by a refined prototypical head, which leverages $f_t^{cl}$ to tweak the prototypes. Finally, we aggregate $V_s$ and $T_s$ to calibrate the two modalities.
• Figure 2: Illustration of the language-guided decoder. The learnable queries are encoded via a multi-head self-attention, and then text features are obtained through two cross-attention modules.
• Figure 3: The schematic overview of the refined prototypical head. The class-level feature is used to adjust the computed prototype.
• Figure 4: The performance of LPN on miniImageNet with N-way K-shot settings. The baseline refers to the results of ProtoNet in our settings.
• Figure 5: The visualization of cross-attention in LaGD on miniImageNet