RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering

TL;DR

This work introduces a novel method known as RSAdapter, which prioritizes runtime and parameter efficiency and improves adaptation to pretrained multimodal models, and allows the parameters of the linear transformation layer to be integrated into the preceding FC layers during inference, reducing inference costs.

Abstract

In recent years, with the rapid advancement of transformer models, transformer-based multimodal architectures have found wide application in various downstream tasks, including but not limited to Image Captioning, Visual Question Answering (VQA), and Image-Text Generation. However, contemporary approaches to Remote Sensing (RS) VQA often involve resource-intensive techniques, such as full fine-tuning of large models or the extraction of image-text features from pre-trained multimodal models, followed by modality fusion using decoders. These approaches demand significant computational resources and time, and a considerable number of trainable parameters are introduced. To address these challenges, we introduce a novel method known as RSAdapter, which prioritizes runtime and parameter efficiency. RSAdapter comprises two key components: the Parallel Adapter and an additional linear transformation layer inserted after each fully connected (FC) layer within the Adapter. This approach not only improves adaptation to pre-trained multimodal models but also allows the parameters of the linear transformation layer to be integrated into the preceding FC layers during inference, reducing inference costs. To demonstrate the effectiveness of RSAdapter, we conduct an extensive series of experiments using three distinct RS-VQA datasets and achieve state-of-the-art results on all three datasets. The code for RSAdapter is available online at https://github.com/Y-D-Wang/RSAdapter.

Figures (6)

• Figure 1: Graphical illustration of the proposed RSAdapter. We insert the RSAdapter next to the MSA and MLP in the transformer block. Among them, the blue block is in the frozen state during training, while the yellow block will be updated. During inference, the weights and biases in the linear transformation can be merged into the preceding fully connected (FC) layer. LT indicates linear transformation.
• Figure 2: Two possible insert positions for RSAdapter.
• Figure 3: Data efficiency comparison with Bi-Modal on LR and HR datasets.
• Figure 4: Data efficiency performance on RSIVQA dataset.
• Figure 5: Typical remote sensing visual question answering examples on LR (a), HR (b) and RSIVQA (c) datasets.