Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding

TL;DR

Calib3D identifies a critical gap in 3D scene understanding: high accuracy does not guarantee reliable uncertainty estimates. It benchmarks 28 3D models across 10 datasets to study aleatoric and epistemic uncertainty, revealing pervasive miscalibration. The paper introduces DeptS, a depth-aware scaling method that adjusts logits based on depth-derived temperature with an entropy-based gate, yielding consistently better calibration than existing baselines. This work provides a rigorous, reproducible framework and practical toolset for developing reliable 3D perception systems in safety-critical settings, with broad implications for autonomous driving and robotics.

Abstract

Safety-critical 3D scene understanding tasks necessitate not only accurate but also confident predictions from 3D perception models. This study introduces Calib3D, a pioneering effort to benchmark and scrutinize the reliability of 3D scene understanding models from an uncertainty estimation viewpoint. We comprehensively evaluate 28 state-of-the-art models across 10 diverse 3D datasets, uncovering insightful phenomena that cope with both the aleatoric and epistemic uncertainties in 3D scene understanding. We discover that despite achieving impressive levels of accuracy, existing models frequently fail to provide reliable uncertainty estimates -- a pitfall that critically undermines their applicability in safety-sensitive contexts. Through extensive analysis of key factors such as network capacity, LiDAR representations, rasterization resolutions, and 3D data augmentation techniques, we correlate these aspects directly with the model calibration efficacy. Furthermore, we introduce DeptS, a novel depth-aware scaling approach aimed at enhancing 3D model calibration. Extensive experiments across a wide range of configurations validate the superiority of our method. We hope this work could serve as a cornerstone for fostering reliable 3D scene understanding. Code and benchmark toolkit are publicly available.

Figures (8)

• Figure 1: Well-calibrated 3D scene understanding models are anticipated to deliver low uncertainties when predictions are accurate and high uncertainties when predictions are inaccurate. Existing 3D models zhu2021cylindrical (UnCal) and prior calibration methods guo2017calibma2021metaCal struggled to provide proper uncertainty estimates. Our proposed depth-aware scaling (DeptS) is capable of outputting accurate estimates, highlighting its potential for real-world usage. The plots shown are the point-wise expected calibration error (ECE) rates. The colormap goes from dark to light, denoting low and high error rates, respectively. Best viewed in colors.
• Figure 2: Depth-correlated patterns in a $\pm50$m LiDAR-acquired scene from the SemanticKITTI behley2019semanticKITTI dataset. (a) Ground truth semantics. (b) Point-wise ECE scores. (c) Point-wise entropy scores.
• Figure 3: The reliability diagrams of visualized calibration gaps from CENet cheng2022cenet on SemanticKITTIbehley2019semanticKITTI. UnCal, TempS, LogiS, MetaC, and DeptS denote the uncalibrated, temperature, logistic, meta, and our depth-aware scaling calibration methods, respectively.
• Figure 4: Ablation studies on (a) relationships between calibration error and intersection-over-union scores, (b) calibration errors of MinkUNet choy2019minkowski using different sparse convolution backends, and (c) average calibration errors of different LiDAR representations.
• Figure 5: Depth-wise confidence and accuracy statistics of uncalibrated (UnCal), temperature scaling (TempS), meta-calibration (MetaC), and our proposed depth-aware scaling (DeptS) methods.