M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data

TL;DR

M3LEO is introduced, a multi-modal, multi-label Earth observation dataset that includes polarimetric, interferometric, and coherence SAR data derived from Sentinel-1, alongside multispectral Sentinel-2 imagery and auxiliary data describing terrain properties such as land use.

Abstract

Satellite-based remote sensing has revolutionised the way we address global challenges. Huge quantities of Earth Observation (EO) data are generated by satellite sensors daily, but processing these large datasets for use in ML pipelines is technically and computationally challenging. While some preprocessed Earth observation datasets exist, their content is often limited to optical or near-optical wavelength data, which is ineffective at night or in adverse weather conditions. Synthetic Aperture Radar (SAR), an active sensing technique based on microwave length radiation, offers a viable alternative. However, the application of machine learning to SAR has been limited due to a lack of ML-ready data and pipelines, particularly for the full diversity of SAR data, including polarimetry, coherence and interferometry. In this work, we introduce M3LEO, a multi-modal, multi-label Earth observation dataset that includes polarimetric, interferometric, and coherence SAR data derived from Sentinel-1, alongside multispectral Sentinel-2 imagery and auxiliary data describing terrain properties such as land use. M3LEO spans approximately 17M 4x4 km data chips from six diverse geographic regions. The dataset is complemented by a flexible PyTorch Lightning framework configured using Hydra to accommodate its use across diverse ML applications in Earth observation. We provide tools to process any dataset available on popular platforms such as Google Earth Engine for seamless integration with our framework. We show that the distribution shift in self-supervised embeddings is substantial across geographic regions, even when controlling for terrain properties. Data: huggingface.co/M3LEO, Code: github.com/spaceml-org/M3LEO.

Figures (9)

• Figure 1: Data splits. Geographic bands for training, validation and test sets, at a ratio of 60:20:20.
• Figure 2: M3LEO dataset and framework. The M3LEO dataset consists of nine ML-ready component datasets and a PyTorch Lightning framework, parameterised by Hydra, for model training.
• Figure 3: Distribution shifts. Distribution shifts between terrain properties described by auxiliary datasets. Distribution shift for the ESAWC categorical data was quantified using L1 distance and continuous data using Wasserstein distance.
• Figure 4: Covariance and appearance shift. (Leftmost) The Sliced Wasserstein Distances (SWD) between reduced train and test set embeddings across AOIs, generated using a masked autoencoder. (Right four) The expected value of same metric conditioned on each of four terrain properties. The expectation was computed with respect to the distribution of the terrain property on the test set. We conditioned on the first three principal components of the ESAWC distribution.
• Figure 5: Embedding scatter plots. Scatter plots of 2D embeddings, reduced using UMAP, coloured according to different auxiliary datasets.