Perceive With Confidence: Statistical Safety Assurances for Navigation with Learning-Based Perception

TL;DR

This work introduces Perceive with Confidence (PwC), a modular calibration framework that uses conformal prediction to bound misdetection in learning-based perception and delivers end-to-end statistical safety assurances for perception-driven navigation. By calibrating pre-trained perception outputs to withstand closed-loop state distribution shifts and combining them with non-deterministic filtering and safety-aware planners, PwC achieves ε-bounded safety with high confidence across unseen environments. The approach is demonstrated with both bounding-box and scene-occupancy predictors, showing substantial reductions in misdetections and improved task success in simulation, and robust hardware validation on a quadruped robot navigating cluttered indoor spaces. PwC’s results underscore the practical impact of lightweight, planner-agnostic uncertainty quantification for safely deploying foundation-model perception in real-world robotic systems, and point to future work on moving agents and multimodal navigation.

Abstract

Rapid advances in perception have enabled large pre-trained models to be used out of the box for transforming high-dimensional, noisy, and partial observations of the world into rich occupancy representations. However, the reliability of these models and consequently their safe integration onto robots remains unknown when deployed in environments unseen during training. To provide safety guarantees, we rigorously quantify the uncertainty of pre-trained perception systems for object detection and scene completion via a novel calibration technique based on conformal prediction. Crucially, this procedure guarantees robustness to distribution shifts in states when perception outputs are used in conjunction with a planner. As a result, the calibrated perception system can be used in combination with any safe planner to provide an end-to-end statistical assurance on safety in unseen environments. We evaluate the resulting approach, Perceive with Confidence (PwC), in simulation and on hardware where a quadruped robot navigates through previously unseen indoor, static environments. These experiments validate the safety assurances for obstacle avoidance provided by PwC. In simulation, our method reduces obstacle misdetection by $70\%$ compared to uncalibrated perception models. While misdetections lead to collisions for baseline methods, our approach consistently achieves $100\%$ safety. We further demonstrate reducing the conservatism of our method without sacrificing safety, achieving a $46\%$ increase in success rates in challenging environments while maintaining $100\%$ safety. In hardware experiments, our method improves empirical safety by $40\%$ over baselines and reduces obstacle misdetection by $93.3\%$. The safety gap widens to $46.7\%$ when navigation speed increases, highlighting our approach's robustness under more demanding conditions.

Figures (56)

• Figure 1: PwC lightly processes the outputs of a pre-trained perception system (green bounding boxes) using conformal prediction in order to ensure a bounded misdetection rate despite any distribution shift in states (gray dots). The calibrated perception system (blue boxes) paired with a non-deterministic filter and a safe planner provide an end-to-end statistical assurance on safety in new test environments.
• Figure 2: Our proposed method is amenable to a range of perception systems, e.g., bounding-box predictors (top), which output a map with predicted bounding boxes (green dashed boxes), and occupancy predictors (bottom), which output a map with predicted occupied regions.
• Figure 3: The configuration space is partitioned into three.
• Figure 4: The filter takes union over the free space.
• Figure 5: Simulation environment in Pybullet.

Theorems & Definitions (2)

• Proposition 1
• Proposition 2