FUSE-RSVLM: Feature Fusion Vision-Language Model for Remote Sensing

TL;DR

This work tackles the core challenge of adapting vision–language models to remote sensing by addressing fine-grained visual fidelity, cross-scale representation, and continual grounding during generation. It introduces MF-RSVLM, a multi-scale feature fusion RS VLM that extracts global context and local details from a high-resolution canvas and injects visual evidence recurrently into a large language model via a gated fusion mechanism. Trained in two stages on a 293K instruction-tuning corpus and evaluated across RS classification, image captioning, and VQA, MF-RSVLM achieves state-of-the-art or highly competitive performance on major RS benchmarks, while mitigating visual forgetting during decoding. The results demonstrate the practical impact of multi-scale fusion and recurrent visual grounding for robust RS understanding and support broader real-world remote sensing applications.

Abstract

Large vision-language models (VLMs) exhibit strong performance across various tasks. However, these VLMs encounter significant challenges when applied to the remote sensing domain due to the inherent differences between remote sensing images and natural images. Existing remote sensing VLMs often fail to extract fine-grained visual features and suffer from visual forgetting during deep language processing. To address this, we introduce MF-RSVLM, a Multi-Feature Fusion Remote Sensing Vision--Language Model that effectively extracts and fuses visual features for RS understanding. MF-RSVLM learns multi-scale visual representations and combines global context with local details, improving the capture of small and complex structures in RS scenes. A recurrent visual feature injection scheme ensures the language model remains grounded in visual evidence and reduces visual forgetting during generation. Extensive experiments on diverse RS benchmarks show that MF-RSVLM achieves state-of-the-art or highly competitive performance across remote sensing classification, image captioning, and VQA tasks. Our code is publicly available at https://github.com/Yunkaidang/RSVLM.

Figures (11)

• Figure 1: Comparison of different models across scene classification, single-image VQA (reported with accuracy), and image captioning (evaluated with METEOR).
• Figure 2: Two-stage training strategy for remote-sensing modalities. In Stage 1 (Pretraining), remote-sensing images are encoded by a CLIP-based vision encoder, projected by an MLP, and fed into the LLM; all modules are trainable. In Stage 2 (Supervised Fine-Tuning), the vision encoder is frozen while the MLP and LLM are further optimized on our instruction-tuning corpus.
• Figure 3: Overview of our method and the architecture of MF-RSVLM, illustrating the multi-scale fusion pipeline. We take as input a low-resolution (336$\times$336) image and a high-resolution (672$\times$672) image. From the high-resolution view, multi-scale sliding windows produce a set of image patches (A), which are processed together with the low-resolution image by a shared, multi-scale vision encoder to construct a high-resolution feature canvas. The resulting feature maps are upsampled to yield multiple groups of visual feature stacks (B). High- and low-resolution features are then concatenated and passed through an MLP to obtain fused detail feature tokens. Finally, a gate injects this fused detail feature into selected hidden layers of the LLM (C).
• Figure 4: The case study of Phi-3.5-Vision, MiniCPM-V-2.6, GeoChat, and our model on three remote sensing task types (category, existence, and counting).
• Figure 5: Examples of single-image multi-turn VQA on highway scenes.