Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation

Thomas Kerdreux, Alexandre Tuel, Quentin Febvre, Alexis Mouche, Bertrand Chapron

TL;DR

The paper tackles the inefficiency and potential bias arising from highly redundant and imbalanced pre-training data in Earth Observation SSL. It introduces a dynamic coreset pruning framework that does not require a pre-existing feature extractor, enabling diverse and balanced SSL pre-training directly on de novo modalities, demonstrated on Sentinel-1 WV ocean SAR data. Through extensive experiments and three downstream tasks, the approach yields faster convergence and improved transferability, and the authors release OceanSAR-1 as a specialized SAR foundation model for ocean observation. This work provides a practical, scalable pathway to robust EO foundation models with reduced compute, particularly for modalities with limited curated datasets.

Abstract

Self-supervised learning (SSL) has enabled the development of vision foundation models for Earth Observation (EO), demonstrating strong transferability across diverse remote sensing tasks. While prior work has focused on network architectures and training strategies, the role of dataset curation, especially in balancing and diversifying pre-training datasets, remains underexplored. In EO, this challenge is amplified by the redundancy and heavy-tailed distributions common in satellite imagery, which can lead to biased representations and inefficient training. In this work, we propose a dynamic dataset pruning strategy designed to improve SSL pre-training by maximizing dataset diversity and balance. Our method iteratively refines the training set without requiring a pre-existing feature extractor, making it well-suited for domains where curated datasets are limited or unavailable. We demonstrate our approach on the Sentinel-1 Wave Mode (WV) Synthetic Aperture Radar (SAR) archive, a challenging dataset dominated by ocean observations. We train models from scratch on the entire Sentinel-1 WV archive spanning 10 years. Across three downstream tasks, our results show that dynamic pruning improves both computational efficiency and representation quality, leading to stronger transferability. We also release the weights of OceanSAR-1, the first model in the OceanSAR family, a series of foundation models for ocean observation and analysis using SAR imagery, at github.com/galeio-research/OceanSAR-models/.

Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation

TL;DR

The paper tackles the inefficiency and potential bias arising from highly redundant and imbalanced pre-training data in Earth Observation SSL. It introduces a dynamic coreset pruning framework that does not require a pre-existing feature extractor, enabling diverse and balanced SSL pre-training directly on de novo modalities, demonstrated on Sentinel-1 WV ocean SAR data. Through extensive experiments and three downstream tasks, the approach yields faster convergence and improved transferability, and the authors release OceanSAR-1 as a specialized SAR foundation model for ocean observation. This work provides a practical, scalable pathway to robust EO foundation models with reduced compute, particularly for modalities with limited curated datasets.

Abstract

Self-supervised learning (SSL) has enabled the development of vision foundation models for Earth Observation (EO), demonstrating strong transferability across diverse remote sensing tasks. While prior work has focused on network architectures and training strategies, the role of dataset curation, especially in balancing and diversifying pre-training datasets, remains underexplored. In EO, this challenge is amplified by the redundancy and heavy-tailed distributions common in satellite imagery, which can lead to biased representations and inefficient training. In this work, we propose a dynamic dataset pruning strategy designed to improve SSL pre-training by maximizing dataset diversity and balance. Our method iteratively refines the training set without requiring a pre-existing feature extractor, making it well-suited for domains where curated datasets are limited or unavailable. We demonstrate our approach on the Sentinel-1 Wave Mode (WV) Synthetic Aperture Radar (SAR) archive, a challenging dataset dominated by ocean observations. We train models from scratch on the entire Sentinel-1 WV archive spanning 10 years. Across three downstream tasks, our results show that dynamic pruning improves both computational efficiency and representation quality, leading to stronger transferability. We also release the weights of OceanSAR-1, the first model in the OceanSAR family, a series of foundation models for ocean observation and analysis using SAR imagery, at github.com/galeio-research/OceanSAR-models/.

Paper Structure

This paper contains 19 sections, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset pruning/coreset selection
  4. Sentinel-1 SAR foundation models
  5. Methodology and Results
  6. Sentinel-1 WV data
  7. Training Framework
  8. Dynamic coreset selection for highly redundant dataset
  9. Experiments
  10. Downstream Tasks
  11. TenGeoP dataset
  12. Significant Wave Height dataset
  13. Wind speed dataset
  14. Results
  15. Impact of dataset pruning
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Illustration of the 10 geophysical classes in the TenGeoP dataset.
  • Figure 2: DINO validation loss as a function of training epoch for a ViT-S/16 model, with (blue) no data pruning, (orange) pruning that discards 50% of the training dataset and (green) pruning that discards 75% of the training dataset. For the latter two curves, black squares indicate epochs at which dataset pruning is performed.
  • Figure 3: Same as Figure \ref{['fig:val_loss']}, but for the k-NN classification accuracy on TenGeoP.