Enhancing Monocular Height Estimation via Weak Supervision from Imperfect Labels

TL;DR

This work tackles monocular height estimation under label scarcity and domain shift by training with imperfect, out-of-domain labels. It introduces an ensemble-based pipeline with a shared encoder and quality-specific branches, a domain classifier to weigh outputs, and loss terms including a balanced soft height loss $L_{BSH}$ and ordinal constraints $L_{OC}$, complemented by a ground truth augmentation module. The method demonstrates that training with mid-/low-quality labels yields better cross-domain generalization, achieving up to 22.94% RMSE reduction on DFC23 and 18.62% on GBH over strong baselines, with ablations confirming each component's contribution. The approach offers a practical path to scale monocular height estimation to regions lacking LiDAR data, with robust performance across diverse urban contexts.

Abstract

Monocular height estimation provides an efficient and cost-effective solution for three-dimensional perception in remote sensing. However, training deep neural networks for this task demands abundant annotated data, while high-quality labels are scarce and typically available only in developed regions, which limits model generalization and constrains their applicability at large scales. This work addresses the problem by leveraging imperfect labels from out-of-domain regions to train pixel-wise height estimation networks, which may be incomplete, inexact, or inaccurate compared to high-quality annotations. We introduce an ensemble-based pipeline compatible with any monocular height estimation network, featuring architecture and loss functions specifically designed to leverage information in noisy labels through weak supervision, utilizing balanced soft losses and ordinal constraints. Experiments on two datasets -- DFC23 (0.5--1 m) and GBH (3 m) -- show that our method achieves more consistent cross-domain performance, reducing average RMSE by up to 22.94% on DFC23 and 18.62% on GBH compared with baselines. Ablation studies confirm the contribution of each design component.

Figures (7)

• Figure 1: Common label qualities for monocular height estimation. Data from BayerischeVermessungsverwaltung_2025_OpenData.
• Figure 2: Network architecture. The proposed ensemble-based network consists of separate branches dedicated to high-, mid-, and low-quality labels. A "domain" classifier is employed to distinguish between label qualities and blend the outputs from each branch into the final prediction. Additionally, outputs from the high-quality branch are used for ground truth augmentation for the mid- and low-quality labels, with backward gradients stopped to prevent interference.
• Figure 3: Height value distribution of the GBH high-quality training and validation set. The distribution exhibits a pronounced long-tailed characteristic: approximately 3e8 pixels (57% of the total) correspond to background regions with heights below 1 m, whereas the number of pixels at large height values decreases sharply to only about 10 per 1 m bin. A similar long-tailed distribution is observed in the building ground-truth height values.
• Figure 4: Distribution of data samples.
• Figure 5: Qualitative results of different models on DFC23. The maps are scaled to the same range. One sample from each test set is visualized per block. H: Backbone network trained on high-quality labels only; M: Backbone network trained on mid-quality labels only; H+M: Backbone network trained on both high-quality and mid-quality labels; Ours H+M: Our proposed pipeline using the backbone network trained on both high- and mid-quality labels.