Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding

Marco Stricker, Masakazu Iwamura, Koichi Kise

TL;DR

This work tackles the challenge of cloud distortion in optical remote sensing by constructing CBEN, a CloudyBigEarthNet dataset that injects realistic cloud occlusions into a large multimodal SAR–optical corpus. It shows that models pretrained on cloud-free data deteriorate significantly on cloudy imagery, yet fine-tuning with cloudy samples restores robustness across both cloud-free and cloudy test conditions. By evaluating self-supervised learning approaches (MAE and MoCo) on ResNet-50 and ViT backbones with early SAR–optical fusion, the study reveals that cloud-aware finetuning yields substantial AP gains (up to ~29 percentage points) and stabilizes performance under occlusion, while highlighting limitations of certain SSL pretraining strategies like MAE in cloudy contexts. The results underscore the practical need to include cloudy data in training for time-sensitive remote sensing tasks and point to future directions in cloud-focused SSL objectives and attention mechanisms for improved cloud resilience.

Abstract

Clouds are a common phenomenon that distorts optical satellite imagery, which poses a challenge for remote sensing. However, in the literature cloudless analysis is often performed where cloudy images are excluded from machine learning datasets and methods. Such an approach cannot be applied to time sensitive applications, e.g., during natural disasters. A possible solution is to apply cloud removal as a preprocessing step to ensure that cloudfree solutions are not failing under such conditions. But cloud removal methods are still actively researched and suffer from drawbacks, such as generated visual artifacts. Therefore, it is desirable to develop cloud robust methods that are less affected by cloudy weather. Cloud robust methods can be achieved by combining optical data with radar, a modality unaffected by clouds. While many datasets for machine learning combine optical and radar data, most researchers exclude cloudy images. We identify this exclusion from machine learning training and evaluation as a limitation that reduces applicability to cloudy scenarios. To investigate this, we assembled a dataset, named CloudyBigEarthNet (CBEN), of paired optical and radar images with cloud occlusion for training and evaluation. Using average precision (AP) as the evaluation metric, we show that state-of-the-art methods trained on combined clear-sky optical and radar imagery suffer performance drops of 23-33 percentage points when evaluated on cloudy images. We then adapt these methods to cloudy optical data during training, achieving relative improvement of 17.2-28.7 percentage points on cloudy test cases compared with the original approaches. Code and dataset are publicly available at: https://github.com/mstricker13/CBEN

CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding

TL;DR

This work tackles the challenge of cloud distortion in optical remote sensing by constructing CBEN, a CloudyBigEarthNet dataset that injects realistic cloud occlusions into a large multimodal SAR–optical corpus. It shows that models pretrained on cloud-free data deteriorate significantly on cloudy imagery, yet fine-tuning with cloudy samples restores robustness across both cloud-free and cloudy test conditions. By evaluating self-supervised learning approaches (MAE and MoCo) on ResNet-50 and ViT backbones with early SAR–optical fusion, the study reveals that cloud-aware finetuning yields substantial AP gains (up to ~29 percentage points) and stabilizes performance under occlusion, while highlighting limitations of certain SSL pretraining strategies like MAE in cloudy contexts. The results underscore the practical need to include cloudy data in training for time-sensitive remote sensing tasks and point to future directions in cloud-focused SSL objectives and attention mechanisms for improved cloud resilience.

Abstract

Clouds are a common phenomenon that distorts optical satellite imagery, which poses a challenge for remote sensing. However, in the literature cloudless analysis is often performed where cloudy images are excluded from machine learning datasets and methods. Such an approach cannot be applied to time sensitive applications, e.g., during natural disasters. A possible solution is to apply cloud removal as a preprocessing step to ensure that cloudfree solutions are not failing under such conditions. But cloud removal methods are still actively researched and suffer from drawbacks, such as generated visual artifacts. Therefore, it is desirable to develop cloud robust methods that are less affected by cloudy weather. Cloud robust methods can be achieved by combining optical data with radar, a modality unaffected by clouds. While many datasets for machine learning combine optical and radar data, most researchers exclude cloudy images. We identify this exclusion from machine learning training and evaluation as a limitation that reduces applicability to cloudy scenarios. To investigate this, we assembled a dataset, named CloudyBigEarthNet (CBEN), of paired optical and radar images with cloud occlusion for training and evaluation. Using average precision (AP) as the evaluation metric, we show that state-of-the-art methods trained on combined clear-sky optical and radar imagery suffer performance drops of 23-33 percentage points when evaluated on cloudy images. We then adapt these methods to cloudy optical data during training, achieving relative improvement of 17.2-28.7 percentage points on cloudy test cases compared with the original approaches. Code and dataset are publicly available at: https://github.com/mstricker13/CBEN
Paper Structure (29 sections, 1 equation, 7 figures, 6 tables)

This paper contains 29 sections, 1 equation, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets in Remote Sensing
  4. Cloudless Machine Learning
  5. Cloud Removal
  6. Cloud Robust Analysis
  7. Data
  8. Remote Sensing Modalities
  9. SSL4EO-S12
  10. BigEarthNet
  11. CBEN -- CloudyBigEarthNet
  12. Methods
  13. Self-Supervised Learning
  14. Masked Autoencoder
  15. Momentum Contrast
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Top row shows optical (RGB channels) of a subarea of the prefecture of Osaka during a cloud free and a cloudy time period. Bottom row shows radar images (VH channel) at a similar date compared to the corresponding image above.
  • Figure 2: Example location of SSL4EO-S12. The columns from left to right represent the timestamps at spring, summer, fall and winter. The rows from top to bottom represent radar, and optical level 1 and level 2 data.
  • Figure 3: Visualization of several samples of BigEarthNet with corresponding labels.
  • Figure 4: Illustration of how BigEarthNet downloaded images. Boxes with literals are called tiles. Boxes with numericals are named patches.
  • Figure 5: Visualization of different tiles and patches of our dataset CBEN with varying cloud cover percentages. (a) Tile with large cloud cover percentage, (b) Tile with low cloud cover percentage, (c) Patch with large cloud cover percentage, extracted from respective tile, (d) Patch with low cloud cover percentage, extracted from respective tile.
  • ...and 2 more figures