Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

TL;DR

This work addresses RSSC by integrating VLM-generated textual descriptions as an auxiliary modality to complement visual features. It introduces a multimodal dual-attention encoder that enables bidirectional cross-attention between image and text, using ViT for images and CLIP for text, with LLaVA generating zero-shot descriptions. Across five diverse RSSC datasets, the proposed model consistently outperforms unimodal and simple fusion baselines, and VLM-generated captions can surpass human annotations in some settings. The results highlight the practical potential of scalable, multimodal fusion for robust RSSC, including zero-shot capabilities and interpretability through attention visualizations, with open avenues for retrieval, temporal/spectral fusion, and segmentation extensions.

Abstract

Remote sensing scene classification (RSSC) is a critical task with diverse applications in land use and resource management. While unimodal image-based approaches show promise, they often struggle with limitations such as high intra-class variance and inter-class similarity. Incorporating textual information can enhance classification by providing additional context and semantic understanding, but manual text annotation is labor-intensive and costly. In this work, we propose a novel RSSC framework that integrates text descriptions generated by large vision-language models (VLMs) as an auxiliary modality without incurring expensive manual annotation costs. To fully leverage the latent complementarities between visual and textual data, we propose a dual cross-attention-based network to fuse these modalities into a unified representation. Extensive experiments with both quantitative and qualitative evaluation across five RSSC datasets demonstrate that our framework consistently outperforms baseline models. We also verify the effectiveness of VLM-generated text descriptions compared to human-annotated descriptions. Additionally, we design a zero-shot classification scenario to show that the learned multimodal representation can be effectively utilized for unseen class classification. This research opens new opportunities for leveraging textual information in RSSC tasks and provides a promising multimodal fusion structure, offering insights and inspiration for future studies. Code is available at: https://github.com/CJR7/MultiAtt-RSSC

Figures (8)

• Figure 1: Examples of challenges in image-only RSSC methods and zero-shot descriptions generated by the LLaVa VLM: (a) High intra-class variance; (b) High inter-class similarity; (c) Large variations in the scale of objects or scenes; (d) Coexistence of multiple ground objects within a single image.
• Figure 2: Illustration of the proposed framework. The RS images are paired with designed prompts and input into VLMs to generate text descriptions. These images are then processed by the Image Encoder to extract image embeddings, while the VLM-generated text is tokenized using the Text Encoder to produce text embeddings. Both sets of embeddings are fed into the Multimodal Dual-Attention Encoder where the complementary information are integrated and enhanced. The fused multimodal representations are then passed through a projection layer to produce the final classification output.
• Figure 3: Cross attention mechanism illustration.
• Figure 4: Comparison of Baseline models and our model. (a) Image embeddings or text embeddings are put into the CNN model for unimodal training; (b) Image and text embeddings are first concatenated and then put into the CNN model; (c) Image embeddings and text embeddings are put into two CNN models separately, the decisions are fused by voting; (d) Our multimodal dual-attention network.
• Figure 5: TSNE visualization of the AID test set.