Mask-to-Height: A YOLOv11-Based Architecture for Joint Building Instance Segmentation and Height Classification from Satellite Imagery

TL;DR

This paper tackles joint building instance segmentation and height classification from satellite imagery by reframing height as a discrete five-class problem. It introduces a YOLOv11-based multitask framework that directly integrates footprint delineation and height tier prediction, using mean DSM values within each building mask to assign height classes. On the DFC2023 Track 2 dataset, the approach achieves strong segmentation performance (e.g., $mAP@50\approx84.2\%$ for footprints) and robust height-class discrimination across five categories (overall $mAP@50(B)\approx61.2\%$, $mAP@50(M)\approx60.4\%$), with particularly good accuracy for rare high-rise classes. The results demonstrate that discrete height modeling enhances interpretability, robustness to label noise, and deployment efficiency, enabling real-time, large-scale semantic urban reconstruction without the need for heavy regression-based height estimation.

Abstract

Accurate building instance segmentation and height classification are critical for urban planning, 3D city modeling, and infrastructure monitoring. This paper presents a detailed analysis of YOLOv11, the recent advancement in the YOLO series of deep learning models, focusing on its application to joint building extraction and discrete height classification from satellite imagery. YOLOv11 builds on the strengths of earlier YOLO models by introducing a more efficient architecture that better combines features at different scales, improves object localization accuracy, and enhances performance in complex urban scenes. Using the DFC2023 Track 2 dataset -- which includes over 125,000 annotated buildings across 12 cities -- we evaluate YOLOv11's performance using metrics such as precision, recall, F1 score, and mean average precision (mAP). Our findings demonstrate that YOLOv11 achieves strong instance segmentation performance with 60.4\% mAP@50 and 38.3\% mAP@50--95 while maintaining robust classification accuracy across five predefined height tiers. The model excels in handling occlusions, complex building shapes, and class imbalance, particularly for rare high-rise structures. Comparative analysis confirms that YOLOv11 outperforms earlier multitask frameworks in both detection accuracy and inference speed, making it well-suited for real-time, large-scale urban mapping. This research highlights YOLOv11's potential to advance semantic urban reconstruction through streamlined categorical height modeling, offering actionable insights for future developments in remote sensing and geospatial intelligence.

Figures (4)

• Figure 1: Structure diagram of the YOLOv11 network
• Figure 2: Training and validation metrics for building instance segmentation. Loss functions (Box Loss, Segmentation Loss, Classification Loss, DFL) are reported for both training and validation sets.
• Figure 3: Training metrics for building instance segmentation (Precision, Recall, mAP@50, mAP@50–95) are reported for training set only
• Figure 4: Qualitative results on DFC2023 validation set: (left) original imagery, (middle) predicted segmentation with height classification, (right) ground truth.