Deep Semantic-Visual Alignment for Zero-Shot Remote Sensing Image Scene Classification

TL;DR

This work addresses zero-shot remote sensing scene classification by automatically generating visually detectable attributes (RSMM-Attributes) via a CLIP-based multi-modal framework and using a Deep Semantic-Visual Alignment (DSVA) architecture. DSVA leverages a Vision Transformer to capture global RS context, a Visual-Attribute Mapping module to project images into an attribute space, and an Attention Concentration mechanism to focus on informative regions, trained with semantic-compatibility and semantic-regression losses. The approach enables generalized ZSL with calibrated stacking and achieves state-of-the-art results on the RSSDIVCS benchmark, significantly outperforming language-based embeddings and previous RS ZSL methods, while reducing manual attribute annotation labor. The method's cross-modal attribute space and interpretable attention maps facilitate robust knowledge transfer to unseen RS categories, supporting scalable, dynamic RS databases.

Abstract

Deep neural networks have achieved promising progress in remote sensing (RS) image classification, for which the training process requires abundant samples for each class. However, it is time-consuming and unrealistic to annotate labels for each RS category, given the fact that the RS target database is increasing dynamically. Zero-shot learning (ZSL) allows for identifying novel classes that are not seen during training, which provides a promising solution for the aforementioned problem. However, previous ZSL models mainly depend on manually-labeled attributes or word embeddings extracted from language models to transfer knowledge from seen classes to novel classes. Besides, pioneer ZSL models use convolutional neural networks pre-trained on ImageNet, which focus on the main objects appearing in each image, neglecting the background context that also matters in RS scene classification. To address the above problems, we propose to collect visually detectable attributes automatically. We predict attributes for each class by depicting the semantic-visual similarity between attributes and images. In this way, the attribute annotation process is accomplished by machine instead of human as in other methods. Moreover, we propose a Deep Semantic-Visual Alignment (DSVA) that take advantage of the self-attention mechanism in the transformer to associate local image regions together, integrating the background context information for prediction. The DSVA model further utilizes the attribute attention maps to focus on the informative image regions that are essential for knowledge transfer in ZSL, and maps the visual images into attribute space to perform ZSL classification. With extensive experiments, we show that our model outperforms other state-of-the-art models by a large margin on a challenging large-scale RS scene classification benchmark.

Figures (10)

• Figure 1: Illustration of zero-shot learning. By learning the semantic properties, e.g., "water", "road", from seen classes such as lake and wetland, the model would be able to recognize the unseen class "park" composing of those semantic properties.
• Figure 2: Attribute vocabulary. We split 98 attributes into 6 groups, i.e., "color", "object presence", "material", "texture", "shape", "functions".
• Figure 3: The attribute annotation process. Left: We show the fine-tune process with a batch of 4 images and the corresponding text descriptions. Right: The inference process of CLIP network where we measure the similarity between the probe image $x_i$ and attributes.
• Figure 4: The main architecture of the DSVA model. The proposed DSVA model consists of a vision transformer that extracts image features, a Visual Attribute Mapping Module that maps the image into attribute space for ZSL classification, and an Attention Concentration Module that focuses on the informative attribute regions.
• Figure 5: Three groups of similar scene categories from the RSSDIVCS li2021learning dataset. We displace three randomly sampled images from each category.