Towards Efficient Benchmarking of Foundation Models in Remote Sensing: A Capabilities Encoding Approach

TL;DR

This work tackles the expensive problem of benchmarking Remote Sensing Vision Foundation Models (RSFMs) across diverse downstream tasks. It introduces capabilities encoding, normalizing per-task performance via $p_{m,t}$ to obtain $\delta_{m,t}$ and $\Delta_{m,t}$, and learning a latent embedding by minimizing $L(x) = \frac{1}{n}\sum (d(x_m,x_t) - \Delta_{m,t})^2$ in a 5-dimensional space using an $L2$ distance. The contributions include a unified database of fine-tuning results, a method to predict downstream performance without full fine-tuning, and latent-space visualizations that reveal trends and central, generalist behavior among RSFMs. This framework enables more efficient benchmarking and provides actionable guidance for dataset curation and model development in Earth observation settings.

Abstract

Foundation models constitute a significant advancement in computer vision: after a single, albeit costly, training phase, they can address a wide array of tasks. In the field of Earth observation, over 75 remote sensing vision foundation models have been developed in the past four years. However, none has consistently outperformed the others across all available downstream tasks. To facilitate their comparison, we propose a cost-effective method for predicting a model's performance on multiple downstream tasks without the need for fine-tuning on each one. This method is based on what we call "capabilities encoding." The utility of this novel approach is twofold: we demonstrate its potential to simplify the selection of a foundation model for a given new task, and we employ it to offer a fresh perspective on the existing literature, suggesting avenues for future research. Codes are available at https://github.com/pierreadorni/capabilities-encoding.

Figures (5)

• Figure 1: An overview of the main RSFMs. The normalized performance is the position in the literature of a model: its value is 1 when the model is the SOTA on the given task, and 0 when it has the worst performance of the literature. See \ref{['sec:norm']} for more details.
• Figure 2: Usage of remote sensing datasets for benchmarking RSFMs in the literature. Most datasets are segmentation or classification datasets, some relate to object detection and change detection, while only one super-resolution dataset is used for benchmarking RSFMs.
• Figure 3: Estimated distance in latent space versus ground truth value $\Delta$ for different distance metrics. L2 and Poincaré metrics perform better than Cosine.
• Figure 4: Effect of the number of fine-tunings of a model in the training data on the prediction accuracy.
• Figure 5: 2-dimensional UMAP visualization of the 5-dimensional embedding space. Multiple clusters can be seen, e.g., the few-shot datasets and ResNet based RSFMs on the top right and top left, the main cluster with most methods on the bottom left, etc.