Robust Vision-Based Runway Detection through Conformal Prediction and Conformal mAP

TL;DR

The paper addresses the challenge of reliable runway detection in vision-based landing systems by applying conformal prediction to provide statistical uncertainty guarantees on localization. It introduces Conformal mAP (C-mAP) and Conformal Average Precision (C-AP) to align standard detection benchmarks with downstream requirements like complete containment of ground-truth boxes, and demonstrates that conformalization can dramatically improve coverage-oriented metrics while preserving strong traditional detection performance. Through experiments on the LARD dataset with YOLOv5-small and YOLOv6-small, the authors show that conformalized detectors attain substantial C-mAP gains (roughly 50–60%) with modest mAP trade-offs, and provide insights into margin, stretch, and IoA/IUO behavior under conformalization. The work offers a practical, certifiable approach to uncertainty quantification in aerospace perception pipelines and lays groundwork for further evaluation in multi-object and real-world certification contexts.

Abstract

We explore the use of conformal prediction to provide statistical uncertainty guarantees for runway detection in vision-based landing systems (VLS). Using fine-tuned YOLOv5 and YOLOv6 models on aerial imagery, we apply conformal prediction to quantify localization reliability under user-defined risk levels. We also introduce Conformal mean Average Precision (C-mAP), a novel metric aligning object detection performance with conformal guarantees. Our results show that conformal prediction can improve the reliability of runway detection by quantifying uncertainty in a statistically sound way, increasing safety on-board and paving the way for certification of ML system in the aerospace domain.

Figures (9)

• Figure 1: Illustration of conformal prediction on the runway detection task. Red: Ground Truth, Blue: YOLO Prediction, Green: Outer Conformalized Box.
• Figure 2: Computation of mAP step-by-step on an example from the LARD dataset. The top image shows the visual setup, while the table illustrates how predictions at different confidence levels affect precision and recall.
• Figure 3: Overview of the Visual Landing System (VLS) pipeline. The system processes an input image through three stages: (1) Object Detection to identify and crop the runway region, (2) Feature Regression to extract key geometric features such as corners, and (3) Pose Estimation to compute the aircraft’s position relative to the runway using the extracted features. Each stage builds upon the previous one, with accurate detection being critical to the reliability of the final pose estimation.
• Figure 4: Pipeline examples illustrating different IoU and IoA outcomes.
• Figure 5: Computation of C-(m)AP step-by-step on an example from the LARD dataset. The top image shows the visual setup, while the table illustrates how predictions at different confidence levels affect precision and recall.