Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality

TL;DR

A comprehensive technical survey on foundation models in RS, offering a brand-new perspective by exploring their evolution from unimodality to multimodality and providing a tutorial-like section to guide researchers, especially beginners, on how to train foundation models in RS and apply them to real-world tasks.

Abstract

Remote sensing (RS) techniques are increasingly crucial for deepening our understanding of the planet. As the volume and diversity of RS data continue to grow exponentially, there is an urgent need for advanced data modeling and understanding capabilities to manage and interpret these vast datasets effectively. Foundation models present significant new growth opportunities and immense potential to revolutionize the RS field. In this paper, we conduct a comprehensive technical survey on foundation models in RS, offering a brand-new perspective by exploring their evolution from unimodality to multimodality. We hope this work serves as a valuable entry point for researchers interested in both foundation models and RS and helps them launch new projects or explore new research topics in this rapidly evolving area. This survey addresses the following three key questions: What are foundation models in RS? Why are foundation models needed in RS? How can we effectively guide junior researchers in gaining a comprehensive and practical understanding of foundation models in RS applications? More specifically, we begin by outlining the background and motivation, emphasizing the importance of foundation models in RS. We then review existing foundation models in RS, systematically categorizing them into unimodal and multimodal approaches. Additionally, we provide a tutorial-like section to guide researchers, especially beginners, on how to train foundation models in RS and apply them to real-world tasks. The survey aims to equip researchers in RS with a deeper and more efficient understanding of foundation models, enabling them to get started easily and effectively apply these models across various RS applications.

Figures (4)

• Figure 1: A general learning paradigm for foundation models in RS: self-supervised pretraining $+$ fine-tuning. For illustration, consider a vision-related example: the model is first pretrained on large amounts of unlabeled RS data, using backbone architectures such as MLPs, CNNs, RNNs, or ViTs. Contrastive and generative (reconstruction) losses are commonly employed during pretraining. The pretrained weights are then transferred to downstream RS tasks, where they can either remain frozen or be fine-tuned (typically with emphasis on decoder weights), enabling applications such as classification, segmentation, detection, regression, and generation.
• Figure 2: A statistical overview of foundation models in RS-related fields. (a) presents a quantitative comparison of unimodal and multimodal foundation models in RS. (b) illustrates the distribution of RS foundation models across different journals using a pie chart. (c) depicts the number of RS foundation models developed by different countries. The top three quantities in the two pie charts are highlighted in red, green, and blue.
• Figure 3: Roadmap of Foundation Models in RS from 2018 to 2025: The evolution from unimodal to multimodal approaches in RS. Unimodal data includes RGB, spectral, and SAR data. Multimodal integration is categorized into homogeneous data (e.g., multi-optical), optical-SAR, and heterogeneous data (e.g., combinations of optical-SAR-spectral-DSM, vision-text, vision-audio, geospatial-vision, time-series, and vision-LMM). Different model categories are color-coded to distinguish various data modalities and integration types, with models arranged in chronological order within each time period (vertical orientation).
• Figure 4: A step-by-step workflow for applying pretrained foundation models in RS downstream tasks. This tutorial-like process begins with model selection, followed by environment configuration and model loading, and concludes with fine-tuning and deploying the chosen foundation models for specific RS applications.