Text-Guided Coarse-to-Fine Fusion Network for Robust Remote Sensing Visual Question Answering

TL;DR

This work tackles robust RSVQA under challenging conditions by integrating optical and SAR data through a text‑guided, coarse‑to‑fine fusion framework named TGFNet. Key innovations include the CFAR module, which progressively localizes question‑relevant regions via Key Region Routing, Multi‑head Cross‑Attention, and Image Feature Enhancement, and the Adaptive Multi‑Expert Fusion (AMEF) that blends Optical, SAR, and Fusion experts with a Regional Quality‑Aware Fusion stage and adaptive weighting. The authors also contribute OSVQA, the first large‑scale optical–SAR RSVQA dataset with 6,008 image pairs and 1,036,694 QA pairs across 16 question types, including modality‑quality questions, enabling comprehensive evaluation. Experiments show TGFNet achieves state‑of‑the‑art performance on OSVQA (OA $65.12\%$, AA $71.89\%$) and demonstrate the benefit of multi‑modal fusion for robustness in cloud‑covered and low‑light remote sensing scenarios.

Abstract

Remote Sensing Visual Question Answering (RSVQA) has gained significant research interest. However, current RSVQA methods are limited by the imaging mechanisms of optical sensors, particularly under challenging conditions such as cloud-covered and low-light scenarios. Given the all-time and all-weather imaging capabilities of Synthetic Aperture Radar (SAR), it is crucial to investigate the integration of optical-SAR images to improve RSVQA performance. In this work, we propose a Text-guided Coarse-to-Fine Fusion Network (TGFNet), which leverages the semantic relationships between question text and multi-source images to guide the network toward complementary fusion at the feature level. Specifically, we develop a Text-guided Coarse-to-Fine Attention Refinement (CFAR) module to focus on key areas related to the question in complex remote sensing images. This module progressively directs attention from broad areas to finer details through key region routing, enhancing the model's ability to focus on relevant regions. Furthermore, we propose an Adaptive Multi-Expert Fusion (AMEF) module that dynamically integrates different experts, enabling the adaptive fusion of optical and SAR features. In addition, we create the first large-scale benchmark dataset for evaluating optical-SAR RSVQA methods, comprising 6,008 well-aligned optical-SAR image pairs and 1,036,694 well-labeled question-answer pairs across 16 diverse question types, including complex relational reasoning questions. Extensive experiments on the proposed dataset demonstrate that our TGFNet effectively integrates complementary information between optical and SAR images, significantly improving the model's performance in challenging scenarios. The dataset is available at: https://github.com/mmic-lcl/. Index Terms: Remote Sensing Visual Question Answering, Multi-source Data Fusion, Multimodal, Remote Sensing, OPT-SAR.

Figures (10)

• Figure 1: RSVQA applications are explored in (a) cloud-covered and (b) low-light scenarios, using optical-SAR image pairs with corresponding question-answer examples. Optical images degrade significantly in these conditions, while SAR images remain robust, highlighting SAR's potential to enhance RSVQA performance in challenging environments. Blue and red indicate regions associated with different questions within the same image pair. The text in parentheses denotes the type of question.
• Figure 2: Examples of state-of-the-art (SOTA) RSVQA models RSVQArsivqaHRVQA are evaluated in (a) cloud-covered and (b) low-light scenarios, relying solely on optical images. Question types are highlighted in green, correct answers are indicated in blue, and incorrect answers in red.
• Figure 3: The overall framework of TGFNet is as follows: First, the CLIP clip model fine-tuned on OSVQA dataset is employed for initial feature extraction from both text and images. Next, we propose the Text-guided Coarse-to-Fine Attention Refinement (CFAR) module, which consists of two identical structures, each comprising KRR, MCA, and IE. This module is designed to focus on the image regions relevant to the given question. To effectively leverage the complementary strengths of SAR, optical, and fusion images for answer prediction, we introduce the Adaptive Multi-Expert Fusion (AMEF) module, which includes the SAR Expert, OPT Expert, Fusion Expert, RQAF, and AF.
• Figure 4: The network structures of MCA and IE are illustrated as follows. Panel (a) shows the structure of the MCA, which consists of two LayerNorm layers, a multi-head cross-attention layer, and a Multi-Layer Perceptron (MLP). It takes key regions of optical or SAR images and the question representation as inputs, outputting their fused results. Panel (b) depicts the structure of the IE, which comprises two LayerNorm layers, a Multi-Head Similarity Enhancement layer, and an MLP. The IE enhances the input optical and SAR images using the output from the MCA, highlighting regions relevant to the question.
• Figure 5: The network structure of RQAF comprises a LayerNorm layer, two linear layers, a multi-head quality-aware patch fusion layer, and an MLP. The question and optical-SAR images are simultaneously fed into the RQAF model, where the high-level semantics of the question guide the quality-aware fusion of the optical and SAR images at each spatial location.