MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining

TL;DR

This work tackles the task-discrepancy between upstream pretraining and downstream RS tasks by proposing Multi-Task Pretraining (MTP) for RS foundation models. Leveraging SAMRS, MTP trains a shared encoder with task-specific decoders to jointly learn semantic segmentation, instance segmentation, and rotated object detection in a single framework, compatible with both CNN and vision transformer backbones exceeding $3\times 10^2$ million parameters. Across 14 RS datasets and multiple tasks, MTP improves performance over comparable-size baselines and remains competitive with larger models, with pronounced gains in low-data finetuning scenarios. The study also analyzes factors like training budget, data volume, and decoder reuse, providing practical guidance for deploying RS foundation models in resource-constrained settings and illustrating broad transferability to scene classification, detection, segmentation, and change detection.

Abstract

Foundation models have reshaped the landscape of Remote Sensing (RS) by enhancing various image interpretation tasks. Pretraining is an active research topic, encompassing supervised and self-supervised learning methods to initialize model weights effectively. However, transferring the pretrained models to downstream tasks may encounter task discrepancy due to their formulation of pretraining as image classification or object discrimination tasks. In this study, we explore the Multi-Task Pretraining (MTP) paradigm for RS foundation models to address this issue. Using a shared encoder and task-specific decoder architecture, we conduct multi-task supervised pretraining on the SAMRS dataset, encompassing semantic segmentation, instance segmentation, and rotated object detection. MTP supports both convolutional neural networks and vision transformer foundation models with over 300 million parameters. The pretrained models are finetuned on various RS downstream tasks, such as scene classification, horizontal and rotated object detection, semantic segmentation, and change detection. Extensive experiments across 14 datasets demonstrate the superiority of our models over existing ones of similar size and their competitive performance compared to larger state-of-the-art models, thus validating the effectiveness of MTP.

Figures (7)

• Figure 1: Comparison of various pretraining methods. (a) cspttov sequentially pretrains a foundational model on both natural and RS images. (b) samrs employs a two-stage pretraining strategy to initialize task-specific decoders (e.g., segmentation) using existing foundational models pretrained on either natural or RS images, preserving the decoder during subsequent finetuning. We extend (b) by incorporating multi-task decoders to enhance the representation capacity of the foundational model, facilitating easy transferability across diverse tasks during finetuning, as depicted in (c).
• Figure 2: The overall pipeline of MTP. Inside MTP, the feature pyramid from the backbone network is fed into multiple decoders for various tasks, including rotated object detection, instance segmentation, and semantic segmentation. These tasks are supervised by diverse labels in the SAMRS dataset. Following MTP, the pretrained model is transferred to different RS tasks for finetuning.
• Figure 3: The finetuning accuracy of different pretrained models with varying training sample sizes. (a) InternImage-XL on EuroSAT. (b) ViT-L + RVSA on SpaceNetv1.
• Figure 4: Visualization of the horizontal object detection predictions of MAE + MTP pretrained ViT-L + RVSA. The images of the first and the second rows are from Xview and DIOR testing sets, respectively.
• Figure 5: Visualization of the rotated object detection predictions of MAE + MTP pretrained ViT-L + RVSA. The images in four rows are from the testing sets of DIOR-R, FAIR1M-2.0, DOTA-V1.0 and DOTA-V2.0, respectively.