REOBench: Benchmarking Robustness of Earth Observation Foundation Models

TL;DR

REOBench introduces the first large-scale robustness benchmark for Earth Observation foundation models, evaluating six core RS tasks under twelve realistic perturbations on high-resolution optical imagery. The study systematically compares MIM-based, CL-based, and vision-language (LLM-based) approaches, revealing substantial degradation under corruptions and clear performance-vs-robustness trends across architectures and tasks. Vision-language and multimodal supervision consistently offer improved resilience, while dimensional trade-offs emerge between backbone size and task granularity. By providing datasets, perturbations, and evaluation protocols, REOBench offers actionable guidance for developing more robust, reliable Earth observation AI for critical applications such as urban planning and disaster response.

Abstract

Earth observation foundation models have shown strong generalization across multiple Earth observation tasks, but their robustness under real-world perturbations remains underexplored. To bridge this gap, we introduce REOBench, the first comprehensive benchmark for evaluating the robustness of Earth observation foundation models across six tasks and twelve types of image corruptions, including both appearance-based and geometric perturbations. To ensure realistic and fine-grained evaluation, our benchmark focuses on high-resolution optical remote sensing images, which are widely used in critical applications such as urban planning and disaster response. We conduct a systematic evaluation of a broad range of models trained using masked image modeling, contrastive learning, and vision-language pre-training paradigms. Our results reveal that (1) existing Earth observation foundation models experience significant performance degradation when exposed to input corruptions. (2) The severity of degradation varies across tasks, model architectures, backbone sizes, and types of corruption, with performance drop varying from less than 1% to over 20%. (3) Vision-language models show enhanced robustness, particularly in multimodal tasks. REOBench underscores the vulnerability of current Earth observation foundation models to real-world corruptions and provides actionable insights for developing more robust and reliable models. Code and data are publicly available at https://github.com/lx709/REOBench.

Figures (9)

• Figure 1: Example of perturbed images. In the first row, we present the original clean image alongside images perturbed by five levels of motion blur. The second and third rows illustrate examples of images corrupted by a range of perturbation types.
• Figure 2: Robustness across different tasks and model architectures. We report the average $\mathcal{R}_{\text{TP}}$ across models.
• Figure 3: Robustness across different backbone sizes. We report the average $\mathcal{R}_{\text{TP}}$ for RemoteCLIP and GeoRSCLIP.
• Figure 4: Robustness across different types of corruptions. We report the $\mathcal{R}_{\text{TP}}$ across models.
• Figure 5: Selected semantic segmentation examples under (a) Gaussian noise (severity 5) and (b) Salt-and-pepper noise (severity 5). For each example, the top row shows the corrupted image and the segmentation results of different models under this corruption, and the bottom row shows the ground truth and segmentation results on clean images.