EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Images

TL;DR

EidMTL, a multitask learning framework that uses evidential heads for depth estimation and semantic segmentation, enabling uncertainty-aware inference from monocular RGB images, is introduced and EvidKimera, an uncertainty-aware semantic surface mapping framework, which uses evidential depth and semantics prediction for improved 3D metric-semantic consistency is presented.

Abstract

For scene understanding in unstructured environments, an accurate and uncertainty-aware metric-semantic mapping is required to enable informed action selection by autonomous systems. Existing mapping methods often suffer from overconfident semantic predictions, and sparse and noisy depth sensing, leading to inconsistent map representations. In this paper, we therefore introduce EvidMTL, a multi-task learning framework that uses evidential heads for depth estimation and semantic segmentation, enabling uncertainty-aware inference from monocular RGB images. To enable uncertainty-calibrated evidential multi-task learning, we propose a novel evidential depth loss function that jointly optimizes the belief strength of the depth prediction in conjunction with evidential segmentation loss. Building on this, we present EvidKimera, an uncertainty-aware semantic surface mapping framework, which uses evidential depth and semantics prediction for improved 3D metric-semantic consistency. We train and evaluate EvidMTL on the NYUDepthV2 and assess its zero-shot performance on ScanNetV2, demonstrating superior uncertainty estimation compared to conventional approaches while maintaining comparable depth estimation and semantic segmentation. In zero-shot mapping tests on ScanNetV2, EvidKimera outperforms Kimera in semantic surface mapping accuracy and consistency, highlighting the benefits of uncertainty-aware mapping and underscoring its potential for real-world robotic applications.

Figures (4)

• Figure 1: Visualization of our evidential multi-task perception pipeline. Given RGB image as input on top left, our EvidMTL framework predicts semantic labels and depth (top) along with their corresponding uncertainty estimates (right). The generated TSDF map, shown in bottom left, from our EvidKimera, leverages these uncertainty measurements, only including cells with low depth uncertainty and assigning unknown labels (grey) to regions with high semantics uncertainty.
• Figure 2: From an input RGB image our proposed EvidMTL model jointly predicts semantics and depth estimates as well as their uncertainty (left part of the figure). The uncertainty estimates correspond well to the error in the prediction compared to the ground truth (GT) as shown on the right.
• Figure 3: Our semantic evidential mapping framework: The RGB image is processed through EvidMTL (left), our evidential multi-task depth-semantic segmentation network. The resulting semantic point cloud undergoes uncertainty-weighted Bayesian fusion for the TSDF layer, whereas the evidential semantic predictions are used as the measurements for updating the voxel semantic priors in EvidKimera (right). The final uncertainty-weighted integration refines the semantic voxel posteriors. The mapping framework outputs metric-semantic information with corresponding uncertainties.
• Figure 4: The columns show the depth error, aleatoric and epistemic uncertainty for a random scene from the ScanNetV2 dataset for the zero-shot evaluation of EvidMTL (EvidSiLog+KL$_{\mu_{n}}$), SiLog+KL$_{\mu_{n}}$, and SiLog+Reg. The error and uncertainty increase from blue to red. The high error red spots on the top are windows. SiLog+KL$_{\mu_{n}}$ (middle row) attributes errors to aleatoric uncertainty whereas SiLog+Reg (bottom row) attributes it to epistemic uncertainty. Our EvidMTL (top row) correctly shows epistemic uncertainty at object boundaries and aleatoric uncertainty on windows and low texture carpets on the floor.