SpaceNet: A Remote Sensing Dataset and Challenge Series

TL;DR

SpaceNet tackles rapid foundational mapping by releasing high-resolution satellite imagery with ground-truth building footprints and road networks, enabling automated feature extraction to support disaster response and humanitarian aid. The paper introduces two task-specific evaluation metrics: a $0.5$ IoU-based $F_1$ for building footprints and the $APLS$ graph-based metric for road networks, including graph augmentation and node snapping techniques. It reports substantial progress across three challenges, from $F_1$ scores of $0.21$--$0.69$ on buildings to a $APLS$ score of $0.66$ for roads, and argues that routing-oriented metrics better capture practical utility than pixel-based scores. The SpaceNet corpus, labeling guidelines, and ongoing challenges aim to accelerate the development of routing-ready geospatial ML and encourage broader use of labeled satellite imagery, with future plans for off-nadir building localization and metadata-informed road travel-time analyses.$IoU$, $F_1$, and $APLS$ are central to the framework and enable comparison across diverse data and algorithms.

Abstract

Foundational mapping remains a challenge in many parts of the world, particularly in dynamic scenarios such as natural disasters when timely updates are critical. Updating maps is currently a highly manual process requiring a large number of human labelers to either create features or rigorously validate automated outputs. We propose that the frequent revisits of earth imaging satellite constellations may accelerate existing efforts to quickly update foundational maps when combined with advanced machine learning techniques. Accordingly, the SpaceNet partners (CosmiQ Works, Radiant Solutions, and NVIDIA), released a large corpus of labeled satellite imagery on Amazon Web Services (AWS) called SpaceNet. The SpaceNet partners also launched a series of public prize competitions to encourage improvement of remote sensing machine learning algorithms. The first two of these competitions focused on automated building footprint extraction, and the most recent challenge focused on road network extraction. In this paper we discuss the SpaceNet imagery, labels, evaluation metrics, prize challenge results to date, and future plans for the SpaceNet challenge series.

Figures (14)

• Figure 1: 200m SpaceNet chip over Rio de Janeiro and attendant building labels
• Figure 2: 400m SpaceNet chip of Las Vegas. Left: SpaceNet GeoJSON road label. Right: RGB image overlaid with road centerlines (orange).
• Figure 3: Legacy metrics often incentivize poor predictions. Left: Ground truth road network in green. Middle: proposal road mask in orange; the road widths are poorly reproduced, though there are no breaks or extraneous connections in the network, yielding scores of F1 = 0.82, relaxed F1 (rF1) = 0.94 for radius = 3), and IoU = 0.69. Right: second proposal road mask in blue; road widths are correct, though a gap exists in the roads (often due to overhanging trees). The right plot yields higher scores (F1 = 0.95, rF1 = 0.96 (radius = 3), and IoU = 0.91). Therefore, legacy metrics prefer the right prediction to the middle prediction, even though the rightmost prediction would not be useful for routing purposes.
• Figure 4: Demonstration of path length difference between sample ground truth and proposal graphs. Left: Shortest path between source (green) and target (red) node in the ground truth graph is shown in yellow, with a path length of $\approx948$ meters. Right: Shortest path between source and target node in the proposal graph with 30 edges removed, with a path length of $\approx1027$ meters; this difference in length forms the basis for our graph similarity metric. Plotting is accomplished via the osmnx python package osmnx.
• Figure 5: Results from the winning implementation for the roads challenge. Top Left: A simple road network in a 400m $\times$ 400m chip from the test set in Las Vegas; the blue line is the ground truth, the yellow line the proposal network, and the APLS score is 0.99. Top Right: A complex road network in Las Vegas; in the center of the graph network there is a disconnect where the divider is located. Bottom Left: A complex network in Shanghai; there are several missed streets in the center of the graph. Bottom Right: A low scoring road network in Khartoum; the proposal network misses several dirt roads, but performs well on the more established paved road network.