NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observation

TL;DR

NeuCo-Bench delivers a task-centric benchmark for evaluating fixed-size neural embeddings in Earth Observation by compressing multi-modal, multi-temporal data into $N$-dimensional vectors $z=E(x)$ and assessing semantic utility through linear probes across diverse tasks. It introduces a hidden-task leaderboard, a size-aware quality score $Q_t^{(p)}$, and a dynamic task-weighted ranking to compare participants under realistic constraints, demonstrated via a data challenge on SSL4EO-S12-derived EO data. The initial results show that multi-modal foundation models, especially when using post-encoding temporal fusion, yield strong semantic performance on land-cover related tasks, while temporal details improve temporally sensitive predictions such as clouds; linear probing remains a robust, scalable evaluation method. The framework, open-source and extensible, aims to standardize EO embedding benchmarks and motivate community contributions across domains beyond EO by focusing on task-driven, compact representations rather than pixel-level reconstruction.

Abstract

We introduce NeuCo-Bench, a novel benchmark framework for evaluating (lossy) neural compression and representation learning in the context of Earth Observation (EO). Our approach builds on fixed-size embeddings that act as compact, task-agnostic representations applicable to a broad range of downstream tasks. NeuCo-Bench comprises three core components: (i) an evaluation pipeline built around reusable embeddings, (ii) a new challenge mode with a hidden-task leaderboard designed to mitigate pretraining bias, and (iii) a scoring system that balances accuracy and stability. To support reproducibility, we release SSL4EO-S12-downstream, a curated multispectral, multitemporal EO dataset. We present initial results from a public challenge at the 2025 CVPR EARTHVISION workshop and conduct ablations with state-of-the-art foundation models. NeuCo-Bench provides a first step towards community-driven, standardized evaluation of neural embeddings for EO and beyond.

Figures (15)

• Figure 1: Workflow diagram: The user of our benchmark compresses a set of downstream data into fixed-length embeddings (size $N=1024$ here). Our benchmark loads the embeddings for each of the $T$ downstream tasks, performs $K$ evaluations, randomly sampling a training and test split from the task data each evaluation, and scores the result compared to all previous experiments as stored in the Scoring Database.
• Figure 2: Spatial distribution of the downstream tasks.
• Figure 3: Quality score $Q_t^{(p)}$ of the participants of the data challenge evaluation phase with corresponding task weight used for ranking. Teams are ordered by their final leaderboard rank, with the winner as Team 1. Team 10 is a simple averaging baseline described in \ref{['sec:eval_general']} and 11 denotes a baseline of normally distributed random embeddings.
• Figure 4: (Left) Radar plot of per‐task $R^2$ performance for a rep. subset of embedding–compression methods. Each axis corresponds to one downstream prediction task; the center denotes $R^2=0$, negative values are clipped to zero for clarity. (Right) Summary of method configurations.
• Figure 5: (a) Average downstream $R^2$ as a function of embedding size $\log N$. Largest size is the full channel count (CNN) or native patch-token dimension (ViT). (b) Average downstream $R^2$ comparing linear probing against one-layer and two-layer MLP probes on 1024-dimensional embeddings.