Scaling Remote Sensing Foundation Models: Data Domain Tradeoffs at the Peta-Scale

TL;DR

This work analyzes scaling RS foundation models trained on petascale EO data, systematically varying data size and model capacity while controlling for architecture and optimization. It demonstrates that RS pretraining under given budgets is primarily data-limited, with a data-scaling law $L(N)=A+B N^{-a}$ where $a\approx0.03$, and shows near-zero gains from increasing model capacity at fixed data via $L(P)=A+B P^{-b}$ with $b\approx0$, while LR scheduling under a Warmup–Stable–Decay regime yields robust convergence. The study also introduces ScaleMAE and MTP along with a scalable G-DAUG weak supervision pipeline, and provides practical guidelines emphasizing data diversity, stable optimization, and fail-fast hyperparameter triage. Overall, the results offer actionable strategies for planning data collection, compute budgets, and optimization schedules to advance frontier-scale RS foundation models and bridge cross-sensor modalities.

Abstract

We explore the scaling behaviors of artificial intelligence to establish practical techniques for training foundation models on high-resolution electro-optical (EO) datasets that exceed the current state-of-the-art scale by orders of magnitude. Modern multimodal machine learning (ML) applications, such as generative artificial intelligence (GenAI) systems for image captioning, search, and reasoning, depend on robust, domain-specialized encoders for non-text modalities. In natural-image domains where internet-scale data is plentiful, well-established scaling laws help optimize the joint scaling of model capacity, training compute, and dataset size. Unfortunately, these relationships are much less well-understood in high-value domains like remote sensing (RS). Using over a quadrillion pixels of commercial satellite EO data and the MITRE Federal AI Sandbox, we train progressively larger vision transformer (ViT) backbones, report success and failure modes observed at petascale, and analyze implications for bridging domain gaps across additional RS modalities. We observe that even at this scale, performance is consistent with a data limited regime rather than a model parameter-limited one. These practical insights are intended to inform data-collection strategies, compute budgets, and optimization schedules that advance the future development of frontier-scale RS foundation models.

Figures (11)

• Figure 1: Visualization of the number of images per 1,000 $\mathrm{km}^2$ in both Akupara-1M and Akupara-100k. The Akupara-100k dataset is subsampled from the Akupara-1M dataset based on maximizing diversity of sensor and land area attributes. Although not visualized, collections over bodies of water, coastlines, islands, and Antarctica were also included in the dataset.
• Figure 2: The Geospatial Data Augmentation (G-DAUG) pipeline leverages OpenStreetMap data and segmentation algorithms to automatically create geospatial data labels. We create pixelwise segmentation masks, as shown, which are also converted bounding boxes and instance segmentation masks for additional supervisory tasks.
• Figure 3: We test eight candidate base learning rates in a fail fast warmup on Akupara-5k, under identical data, batch size, and optimizer settings on four DGX nodes. Each trajectory ramps toward its labeled goal stable learning rate; candidates that exhibit loss spikes, oscillation, or rising gradient norms in this horizon are rejected, while those showing smooth loss decrease and bounded gradient statistics advance to full WSD training.
• Figure 4: We train three selected stable learning rates with the Warmup, Stable, Decay (WSD) schedule on Akupara-10k, comprising a 10% warmup, an 80% stable plateau at the target rate, and an exponential decay initiated when approximately 10% of the data remains. The curves trace the full WSD cycle for each rate, illustrating the transition from warmup to plateau (listed stable LR) and subsequent decay.
• Figure 5: Batch size scaling results show an increase in batch size does not result in increased compute efficiency during early-stage training, based on average loss over three trials. Based on the similar number of steps needed to reach convergence for the learning rates tested, we hypothesize that smaller batch sizes would maneuver this training configuration out of the saturated range and decrease forward pass compute efficiency.