How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

TL;DR

The paper investigates whether remote sensing foundation models follow CV scaling trends or exhibit a different regime. By applying post-hoc width slimming across six RS FMs and four tasks, it demonstrates that RS models maintain substantial accuracy at very low compute (often 70–85% retention at 1% compute), far outperforming CV baselines and supporting the hypothesis of representational redundancy. It further shows that learned slimmable training can boost MoCo-based models, while MAE-based slimmability yields task-dependent results, and provides mechanistic explanations via EVR and feature correlation analyses of architecture-induced scaling strategies. The findings offer a practical pathway for deploying RS FMs under resource constraints and invite a shift away from naive scaling toward deployment-aware, scalable RS modeling.

Abstract

Large-scale foundation models (FMs) in remote sensing (RS) are developed based on the paradigms established in computer vision (CV) and have shown promise for various Earth observation applications. However, the direct transfer of scaling assumptions from CV to RS has not been adequately examined. We hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, where increasing parameter count primarily induces redundant representations rather than qualitatively new abstractions. To test this hypothesis, we use post-hoc slimming, where we uniformly reduce the width of pretrained encoder, as a tool to measure representational redundancy across six state-of-the-art RS FMs on four downstream classification tasks. Our findings reveal a significant contrast with those in the CV domain: while a post-hoc slimmed masked autoencoder (MAE) trained on ImageNet retains less than 10% accuracy at 1% FLOPs, RS FMs maintain over 71% relative accuracy at the same budget. This sevenfold difference provides strong empirical support for our hypothesis. We further demonstrate that learned slimmable training can improve both Momentum Contrast (MoCo)- and MAE- based models. In addition, through the explained variance ratio and the feature correlation analysis, we provide mechanistic explanations showing that RS FMs distribute task-relevant information with high redundancy. Our findings establish post-hoc slimmability as both a practical deployment strategy for resource-constrained environments and a diagnostic tool that challenges the prevailing scaling paradigm in RS. Upon acceptance, we will publish all code.

Figures (13)

• Figure 1: Averaged relative retention rate of six RS FM across four datasets with respect to their computational budget when slimmed post-hoc in comparison with a CV MAE baseline. RS FM retain over 70% relative accuracy at all slimming scales for both KNN and linear probing, whereas a MAE trained on ImageNet-1k imagenet15russakovsky and evaluated on ImageNet-100 tian2020contrastive exhibits sharp degradation.
• Figure 2: KNN classification performance with respect to their relative compute requirements of six pretrained RS FM on: a) m-brick-kiln; b) m-eurosat; c) m-so2sat; and d) m-bigearthnet from the geobench benchmark collection.
• Figure 3: KNN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-eurosat; and b) m-bigearthnet
• Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench benchmark collection with respect to their relative compute requirements on pretrained RS FM: a) DOFA (large); b) TerraMind-1.0 (large); and c) Prithvi-EO-2.0 (600M).
• Figure 5: EVR with respect to the singular value rank index for three RS FM on m-eurosat from the geobench benchmark collection at three relative compute requirements: a) 0.33; b) 0.67; and c) 1.0.