THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applications

TL;DR

THOR introduces a compute-adaptive, multi-sensor foundation model for Earth observation that unifies Sentinel-1, -2, and -3 data across 10 m to 1000 m GSD and enables inference-time trade-offs between patch density and compute without retraining. Central to THOR is a per-band patch projection ViT encoder, a FlexiViT-inspired random patching scheme with a GSD-aware ALiBi encoding, and a lightweight Conv2D-Transpose decoder trained with a multi-task MAE-based objective and land-cover/climate pretext tasks. The THOR Pretrain dataset of 22 TB and its multi-task loss (MAE, contrastive, map prediction, ERA5) yield data-efficient representations that outperform baselines in data-scarce regimes and scale effectively with more labeled data, while enabling test-time deployment with variable patch sizes. The work demonstrates practical impact for climate and society applications by enabling high-resolution detail when needed and scalable, global assessments for coarser tasks, with snow-cover and S3 benchmarks illustrating the gains from flexible decoding and dense token representations.

Abstract

Current Earth observation foundation models are architecturally rigid, struggle with heterogeneous sensors and are constrained to fixed patch sizes. This limits their deployment in real-world scenarios requiring flexible computeaccuracy trade-offs. We propose THOR, a "computeadaptive" foundation model that solves both input heterogeneity and deployment rigidity. THOR is the first architecture to unify data from Copernicus Sentinel-1, -2, and -3 (OLCI & SLSTR) satellites, processing their native 10 m to 1000 m resolutions in a single model. We pre-train THOR with a novel randomized patch and input image size strategy. This allows a single set of pre-trained weights to be deployed at inference with any patch size, enabling a dynamic trade-off between computational cost and feature resolution without retraining. We pre-train THOR on THOR Pretrain, a new, large-scale multi-sensor dataset and demonstrate state-of-the-art performance on downstream benchmarks, particularly in data-limited regimes like the PANGAEA 10% split, validating that THOR's flexible feature generation excels for diverse climate and society applications.

Figures (15)

• Figure 1: THOR encoder uses a single ViT. Data is processed using a band-wise patch projection layer and group average pooling.
• Figure 2: GSD-aware 2D-ALiBi for two groups: 10m GSD and 8x8 patches and 20m GSD and 4x4 patches. Left: Each sub-square is the ALiBi values (Eq. \ref{['eq:alibi']}) between a 20 m GSD patch and all 10 m GSD patches. Mid: Each sub-square is the ALiBi values (Eq. \ref{['eq:alibi']}) between a 10 m GSD patch and all 20 m GSD patches. Right: Full GSD-aware 2D-ALiBi matrix for the full sequence of 8x8 + 4x4 = 80 patches, where the off-diagonal and diagonal blocks are the intra product and inter product attention biases, respectively.
• Figure 3: Pretext tasks used for learning THOR.
• Figure 4: Test mIoU results for THOR-B model with varying patch sizes using a fixed number of tokens equal to 18 with linear probing segmentation on the Sen1Floods11 dataset using Sentinel 1 and Sentinel 2 data, 10% of the training data, with mean aggregation of features.
• Figure 5: Test mIoU results of a THOR-B (w/UperNet), trained on a $108 \times 108$ image size, evaluated on increasingly larger images.