Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers

Wenwen Liao, Hang Ruan

TL;DR

EfficientFSL presents a query-only fine-tuning paradigm for few-shot classification with Vision Transformers by freezing the backbone and injecting lightweight modules that generate task-specific queries, fuse multi-layer features, and align support/query distributions. The Forward Block, Combine Block, and SQ Attention Block collectively enable strong performance with a small number of trainable parameters, demonstrated across four in-domain and six cross-domain benchmarks. Key findings include substantial accuracy gains over SOTA methods, superior parameter efficiency, and ablation-backed validation of each module’s contribution, particularly the SQ Attention Block’s mitigation of distribution shift. The approach offers a practical, scalable solution for deploying ViT-based FSL in low-resource settings, with clear implications for efficient PETL in vision tasks.

Abstract

Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification, owing to their powerful representational capacity. However, fine-tuning such large models demands extensive GPU memory and prolonged training time, making them impractical for many real-world low-resource scenarios. To bridge this gap, we propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT, which achieves competitive performance while significantly reducing computational overhead. EfficientFSL fully leverages the knowledge embedded in the pre-trained model and its strong comprehension ability, achieving high classification accuracy with an extremely small number of tunable parameters. Specifically, we introduce a lightweight trainable Forward Block to synthesize task-specific queries that extract informative features from the intermediate representations of the pre-trained model in a query-only manner. We further propose a Combine Block to fuse multi-layer outputs, enhancing the depth and robustness of feature representations. Finally, a Support-Query Attention Block mitigates distribution shift by adjusting prototypes to align with the query set distribution. With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets, demonstrating its effectiveness in real-world applications.

EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers

TL;DR

EfficientFSL presents a query-only fine-tuning paradigm for few-shot classification with Vision Transformers by freezing the backbone and injecting lightweight modules that generate task-specific queries, fuse multi-layer features, and align support/query distributions. The Forward Block, Combine Block, and SQ Attention Block collectively enable strong performance with a small number of trainable parameters, demonstrated across four in-domain and six cross-domain benchmarks. Key findings include substantial accuracy gains over SOTA methods, superior parameter efficiency, and ablation-backed validation of each module’s contribution, particularly the SQ Attention Block’s mitigation of distribution shift. The approach offers a practical, scalable solution for deploying ViT-based FSL in low-resource settings, with clear implications for efficient PETL in vision tasks.

Abstract

Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification, owing to their powerful representational capacity. However, fine-tuning such large models demands extensive GPU memory and prolonged training time, making them impractical for many real-world low-resource scenarios. To bridge this gap, we propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT, which achieves competitive performance while significantly reducing computational overhead. EfficientFSL fully leverages the knowledge embedded in the pre-trained model and its strong comprehension ability, achieving high classification accuracy with an extremely small number of tunable parameters. Specifically, we introduce a lightweight trainable Forward Block to synthesize task-specific queries that extract informative features from the intermediate representations of the pre-trained model in a query-only manner. We further propose a Combine Block to fuse multi-layer outputs, enhancing the depth and robustness of feature representations. Finally, a Support-Query Attention Block mitigates distribution shift by adjusting prototypes to align with the query set distribution. With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets, demonstrating its effectiveness in real-world applications.
Paper Structure (30 sections, 11 equations, 4 figures, 6 tables)

This paper contains 30 sections, 11 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Few-Shot Learning
  4. Parameter-Efficient Transfer Learning
  5. Preliminaries
  6. Problem Setting
  7. Prototypical Network
  8. Methodology
  9. Overview
  10. Forward Block
  11. Active Block.
  12. Frozen Block.
  13. Combine Block
  14. Feature Alignment using a Shared Combine MLP.
  15. Adaptive Feature Aggregation.
  16. ...and 15 more sections

Figures (4)

  • Figure 1: An overview of our pipeline. EfficientFSL takes intermediate representations from a pre-trained model as input and integrates task-specific knowledge through the Forward Block, Combine Block, and SQ Attention Block, followed by classification using a PN classifier.
  • Figure 2: Cross-domain Results under Different Shots.
  • Figure 3: t-SNE visualization of a 5-way 1-shot 15-query task on the FC100 dataset. Squares represent prototypes computed by simply averaging support features ($s^{c}$); triangles represent prototypes computed by SQ Attention Block ($s^{att}$); circles denote query samples.
  • Figure 4: t-SNE visualizations of test query samples under six settings. (b)-(f) incrementally add components to the previous setting.