Foundation Models for Generalist Geospatial Artificial Intelligence

TL;DR

The paper presents a framework to construct geospatial foundation models and introduces Prithvi, a 100M-parameter transformer pretrained on over 1TB of HLS imagery. It demonstrates strong data-efficient fine-tuning across four Earth observation tasks (cloud gap imputation, flood mapping, wildfire scar segmentation, and crop segmentation) and shows competitive performance against state-of-the-art methods. Key contributions include a scalable data-preprocessing pipeline, a 3D spatiotemporal MAE-based pretraining approach, and open-source release of both weights and fine-tuning workflows. The work underscores the potential of foundation-model paradigms to reduce labeled-data needs while generalizing across regions and resolutions, accelerating geoscience research and applications.

Abstract

Significant progress in the development of highly adaptable and reusable Artificial Intelligence (AI) models is expected to have a significant impact on Earth science and remote sensing. Foundation models are pre-trained on large unlabeled datasets through self-supervision, and then fine-tuned for various downstream tasks with small labeled datasets. This paper introduces a first-of-a-kind framework for the efficient pre-training and fine-tuning of foundational models on extensive geospatial data. We have utilized this framework to create Prithvi, a transformer-based geospatial foundational model pre-trained on more than 1TB of multispectral satellite imagery from the Harmonized Landsat-Sentinel 2 (HLS) dataset. Our study demonstrates the efficacy of our framework in successfully fine-tuning Prithvi to a range of Earth observation tasks that have not been tackled by previous work on foundation models involving multi-temporal cloud gap imputation, flood mapping, wildfire scar segmentation, and multi-temporal crop segmentation. Our experiments show that the pre-trained model accelerates the fine-tuning process compared to leveraging randomly initialized weights. In addition, pre-trained Prithvi compares well against the state-of-the-art, e.g., outperforming a conditional GAN model in multi-temporal cloud imputation by up to 5pp (or 5.7%) in the structural similarity index. Finally, due to the limited availability of labeled data in the field of Earth observation, we gradually reduce the quantity of available labeled data for refining the model to evaluate data efficiency and demonstrate that data can be decreased significantly without affecting the model's accuracy. The pre-trained 100 million parameter model and corresponding fine-tuning workflows have been released publicly as open source contributions to the global Earth sciences community through Hugging Face.

Figures (10)

• Figure 1: We propose a first-of-its-kind framework for the development of geospatial foundation models from raw satellite imagery, which we leverage to generate the Prithvi-100M model. The framework encompasses (1) the sampling, filtering, and pre-processing of raw geospatial data and the self-supervised foundation model pretraining, (2) the fine-tuning to specific downstream applications, and (3) the inference process.
• Figure 2: Geo-regions from the contiguous U.S. are clustered into one of 20 different categories based on temperature and precipitation data.
• Figure 3: The masked autoencoder (MAE) structure for pre-training Prithvi on large-scale multi-temporal and multi-spectral satellite images.
• Figure 4: Pre-training and fine-tuning in Prithvi for various types of downstream tasks.
• Figure 5: Pretraining results of Prithvi using 1TB of HLS data from the contiguous US.