Co-Training Vision Language Models for Remote Sensing Multi-task Learning

TL;DR

RSCoVLM presents a unified remote sensing vision-language model for multi-task learning, addressing data heterogeneity, multi-scale imagery, and fair evaluation. It introduces a data curation engine, a dynamic resolution strategy with a Zoom-in Chain for UHR imagery, and an autoregressive aerial detection framework with confidence-free evaluation metrics. The approach achieves state-of-the-art results across RS tasks including large-scale imagery understanding, grounding, detection, scene classification, and VQA, while remaining fully open-source for reproducibility. This work advances general-purpose RS modeling by integrating diverse tasks under a single flexible VLM, enabling scalable and robust RS reasoning in practical settings.

Abstract

With Transformers achieving outstanding performance on individual remote sensing (RS) tasks, we are now approaching the realization of a unified model that excels across multiple tasks through multi-task learning (MTL). Compared to single-task approaches, MTL methods offer improved generalization, enhanced scalability, and greater practical applicability. Recently, vision language models (VLMs) have achieved promising results in RS image understanding, grounding, and ultra-high-resolution (UHR) image reasoning, respectively. Moreover, the unified text-based interface demonstrates significant potential for MTL. Hence, in this work, we present RSCoVLM, a simple yet flexible VLM baseline for RS MTL. Firstly, we create the data curation engine, including data acquisition, offline processing and integrating, as well as online loading and weighting. This data engine effectively addresses complex RS data enviroment and generates flexible vision-language conversations. Furthermore, we propose a unified dynamic-resolution strategy to address the diverse image scales inherent in RS imagery. For UHR images, we introduce the Zoom-in Chain mechanism together with its corresponding dataset, LRS-VQA-Zoom. The strategies are flexible and effectively mitigate the computational burdens. Additionally, we significantly enhance the model's object detection capability and propose a novel evaluation protocol that ensures fair comparison between VLMs and conventional detection models. Extensive experiments demonstrate that RSCoVLM achieves state-of-the-art performance across diverse tasks, outperforming existing RS VLMs and even rivaling specialized expert models. All the training and evaluating tools, model weights, and datasets have been fully open-sourced to support reproducibility. We expect that this baseline will promote further progress toward general-purpose RS models.

Figures (6)

• Figure 1: Comparisons with existing MTL methods across the resolutions of input images, network architectures, and supported tasks (i.e., detection, grounding, description, and classification).
• Figure 2: Overall schematic diagram of the proposed method. The overall RS MTL framework based on VLM is presented in Section \ref{['sec:framework']}. The data curation engine is introduced in Section \ref{['sec:data']}. The dynamic resolution strategy is proposed in Section \ref{['sec:resolution']}. We introduce the proposed Zoom-in Chain strategy and the corresponding LRS-VQA-Zoom dataset in Section \ref{['zoomin']}. Finally, we describe the aerial detection scheme and propose the fair metric $\text{AP}_{\text{nc}}$ in Section \ref{['sec:detection']}.
• Figure 3: Schematic diagram of the native resolution input.
• Figure 4: Examples of the three types of annotated data in the proposed LRS-VQA-Zoom.
• Figure 5: The impact of confidence scores on $\text{mAP}_{\text{nc}}$ with error bands. The colored lines record the variation trends of $\text{mAP}_{\text{nc}}$ for the popular conventional detector on DOTA-v1.0 dota (trained and evaluated on both the 'train' split and the 'validation' split) dataset under different confidence thresholds.