S-EO: A Large-Scale Dataset for Geometry-Aware Shadow Detection in Remote Sensing Applications

TL;DR

The paper tackles the scarcity of large-scale geometry-aware shadow data for remote sensing and 3D reconstruction. It introduces S-EO, a large, high-resolution dataset with 702 georeferenced 500×500 m tiles across the USA, containing multi-date WorldView-3 PAN/RGB imagery and LiDAR-derived DSMs, plus automatically generated shadow and vegetation masks and RPCs. A shadow-detection model is trained on S-EO using Min/Max DSM variants to mitigate bias, and the approach generalizes to unseen aerial data after limited AISD fine-tuning; moreover, shadow supervision is incorporated into EO-NeRF, improving altitude accuracy and geometry. Overall, S-EO enables robust shadow-aware learning for EO tasks and demonstrates practical benefits for geometry-aware 3D reconstruction, with the dataset, model, and shadow-supervised EO-NeRF pipeline released for public use.

Abstract

We introduce the S-EO dataset: a large-scale, high-resolution dataset, designed to advance geometry-aware shadow detection. Collected from diverse public-domain sources, including challenge datasets and government providers such as USGS, our dataset comprises 702 georeferenced tiles across the USA, each covering 500x500 m. Each tile includes multi-date, multi-angle WorldView-3 pansharpened RGB images, panchromatic images, and a ground-truth DSM of the area obtained from LiDAR scans. For each image, we provide a shadow mask derived from geometry and sun position, a vegetation mask based on the NDVI index, and a bundle-adjusted RPC model. With approximately 20,000 images, the S-EO dataset establishes a new public resource for shadow detection in remote sensing imagery and its applications to 3D reconstruction. To demonstrate the dataset's impact, we train and evaluate a shadow detector, showcasing its ability to generalize, even to aerial images. Finally, we extend EO-NeRF - a state-of-the-art NeRF approach for satellite imagery - to leverage our shadow predictions for improved 3D reconstructions.

Figures (9)

• Figure 1: Example multi-date satellite images and shadow masks of an area from the S-EO dataset. Shadows and their variations are critical cues for Earth Observation algorithms focused on geometry estimation and scene understanding.
• Figure 2: Top to bottom: S-EO dataset images from three sites— in Jacksonville, Omaha, and San Diego. For each site, left to right: the pansharpened RGB image; the DSM computed from the minimum elevation per grid cell (DSM Min) and its derived shadow mask; and the DSM computed from the maximum elevation per grid cell (DSM Max) with its respective shadow mask. DSM Min yields a cleaner shadow mask by filtering out transient elements (e.g., trees), although it may erode building edges and produce smaller shadows. In contrast, DSM Max preserves more building details and generates larger shadows but with increased noise in the shadow mask.
• Figure 3: Limitations of our shadow annotation method. Left to right: pansharpened RGB image, shadow annotations, model predictions (S-EO-trained), and an overlay (magenta: ground truth, cyan: predictions, orange: matches). DSM holes in the cross-shaped building cause false positives, which the model corrects.
• Figure 4: Impact of bias reduction when training with DSM Min and DSM Max shadow masks. The image shows shadow predictions for a San Diego tile: magenta (DSM Min-only training), cyan (Min-Max training), and orange (overlap, where both models agree). Magenta regions extend onto buildings, while cyan aligns better with actual shadows.
• Figure 5: Qualitative results on the S-EO dataset. Top to bottom: Jacksonville (JAX_334_0), Omaha (OMA_93_15), San Diego (UCSD_353_10). Left to right: input image, ground-truth shadow mask, and model prediction.