3D Building Reconstruction from Monocular Remote Sensing Images with Multi-level Supervisions

TL;DR

This work tackles 3D building reconstruction from monocular remote sensing images under limited 3D supervision by introducing MLS-BRN, a multi-level supervised network. It adds two modules, Pseudo Building Bbox Calculator (PBC) and Roof-Offset guided Footprint Extractor (ROFE), along with a height prediction head, enabling effective learning from datasets that provide footprints alone or with partial 3D annotations. A three-level training strategy leverages samples with varying supervision to improve footprint accuracy, height estimation, and overall 3D reconstruction, achieving competitive results across cross-city datasets and significantly reducing reliance on full 3D labels. The approach demonstrates strong footprint extraction, accurate height estimation, and robust cross-city generalization, supporting scalable, cost-efficient large-area city modeling.

Abstract

3D building reconstruction from monocular remote sensing images is an important and challenging research problem that has received increasing attention in recent years, owing to its low cost of data acquisition and availability for large-scale applications. However, existing methods rely on expensive 3D-annotated samples for fully-supervised training, restricting their application to large-scale cross-city scenarios. In this work, we propose MLS-BRN, a multi-level supervised building reconstruction network that can flexibly utilize training samples with different annotation levels to achieve better reconstruction results in an end-to-end manner. To alleviate the demand on full 3D supervision, we design two new modules, Pseudo Building Bbox Calculator and Roof-Offset guided Footprint Extractor, as well as new tasks and training strategies for different types of samples. Experimental results on several public and new datasets demonstrate that our proposed MLS-BRN achieves competitive performance using much fewer 3D-annotated samples, and significantly improves the footprint extraction and 3D reconstruction performance compared with current state-of-the-art. The code and datasets of this work will be released at https://github.com/opendatalab/MLS-BRN.git.

Figures (16)

• Figure 1: Our proposed method achieves 3D building reconstruction by training samples of different annotation levels. Large quantity of samples only include building footprint annotations, whereas a small quantity of samples contain extra roof-to-footprint offset and building height annotations.
• Figure 2: An overview of our proposed method. Taking a monocular remote sensing image as input, our MLS-BRN generates a set of building bboxes, roof-to-footprint offsets, building heights, and pixel-wise roof masks. The predicted roof masks and their corresponding offsets are further integrated to predict pixel-wise footprint masks. The predicted footprint mask and building height are used to produce the final vectorized 3D model. Two novel modules are introduced: (1) the ROFE predicts footprint masks guided by the predicted roof masks and offsets; (2) the PBC predicts off-nadir and offset angles to calculate pseudo building bboxes for building bbox-unknown samples.
• Figure 3: The results of the baselines and our method trained on $BN_{100}$ and tested on the BONAI test set in terms of the footprint segmentation performance. The yellow, cyan, and red polygons denote the TP, FP, and FN.
• Figure 4: The visualization results of building height prediction from our method and LOFT-FOA+H on the OmniCity-view3 test set.
• Figure 5: 3D reconstruction results of Shanghai, Xi'an, Hong Kong, and New York obtained using our method. The remote sensing images for Shanghai and Xi'an are chosen from the BONAI test set, whereas the remote sensing image for New York is chosen from the OmniCity-view3 test set.