Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification

TL;DR

This work tackles the challenging problem of remote-sensing fine-grained ship classification (RS-FGSC) under limited labeled data and unseen classes. It introduces a hierarchical, multi-granularity prompt-tuning framework that augments a large vision-language model with remote-sensing priors via a lightweight bias network, while keeping the CLIP encoders fixed. A new FGSCM-52 dataset is proposed to benchmark base-to-new generalization in RS-FGSC with expanded categories and detailed annotations. Extensive experiments across FGSC-23, FGSCR-42, EuroSAT, and FGSCM-52 demonstrate state-of-the-art performance, with notable gains in base-to-new harmonic means and improved attention focus on ship targets. The approach offers practical benefits for real-world RS-FGSC deployment, particularly where labeled data are scarce or privacy constraints limit data collection.

Abstract

Fine-grained ship classification in remote sensing (RS-FGSC) poses a significant challenge due to the high similarity between classes and the limited availability of labeled data, limiting the effectiveness of traditional supervised classification methods. Recent advancements in large pre-trained Vision-Language Models (VLMs) have demonstrated impressive capabilities in few-shot or zero-shot learning, particularly in understanding image content. This study delves into harnessing the potential of VLMs to enhance classification accuracy for unseen ship categories, which holds considerable significance in scenarios with restricted data due to cost or privacy constraints. Directly fine-tuning VLMs for RS-FGSC often encounters the challenge of overfitting the seen classes, resulting in suboptimal generalization to unseen classes, which highlights the difficulty in differentiating complex backgrounds and capturing distinct ship features. To address these issues, we introduce a novel prompt tuning technique that employs a hierarchical, multi-granularity prompt design. Our approach integrates remote sensing ship priors through bias terms, learned from a small trainable network. This strategy enhances the model's generalization capabilities while improving its ability to discern intricate backgrounds and learn discriminative ship features. Furthermore, we contribute to the field by introducing a comprehensive dataset, FGSCM-52, significantly expanding existing datasets with more extensive data and detailed annotations for less common ship classes. Extensive experimental evaluations demonstrate the superiority of our proposed method over current state-of-the-art techniques. The source code will be made publicly available.

Figures (5)

• Figure 1: Visualization of the attention maps of (a) the baseline CLIP clip and (b) the proposed method. Our method concentrates on ship targets while the CLIP model attends to the background excessively.
• Figure 2: Overview of the proposed method. The main idea is that the model leverages a hierarchical, multi-granularity prompt design and incorporates remote sensing ship priors to learn generalizable feature representation.
• Figure 3: The different between our method and CLIP in the Text Encoder.
• Figure 4: Overview of the difference between our FGSCM-52 dataset and the original FGSCR-42 dataset. Sample quantities are presented on a logarithmic scale.
• Figure 5: Visualization of the attention map of different methods. In each group, the ship image, the attention map of the CLIP method, and the attention map of the proposed method are displayed from left to right.