ObjectTransforms for Uncertainty Quantification and Reduction in Vision-Based Perception for Autonomous Vehicles

TL;DR

This work tackles uncertainty in vision-based autonomous vehicle perception by introducing ObjectTransforms, which apply object-specific perturbations during training (HSV color-space perturbations and diffusion-based pedestrian augmentation) and at inference time (controlled HSV perturbations) to quantify and reduce predictive uncertainty. The authors formalize uncertainty as a violation of transformation invariance and define a variance-based metric $U(C)$ composed of $U_{class}(C)$ and $U_{bbox}(C)$, enabling uncertainty-driven filtering and threshold adjustment. Empirically, using YOLOv8 on nuImages 10K, ObjectTransforms improves $\mathrm{mAP}_{50-95}$ during training and provides a real-time uncertainty signal that reduces false positives (and can recover false negatives) at inference, achieving practical 5 fps on a modest GPU. The results suggest that object-level, test-time transformations offer a lightweight yet effective mechanism for uncertainty quantification and reduction in AV perception, with potential for broader deployment and future work in adverse conditions.

Abstract

Reliable perception is fundamental for safety critical decision making in autonomous driving. Yet, vision based object detector neural networks remain vulnerable to uncertainty arising from issues such as data bias and distributional shifts. In this paper, we introduce ObjectTransforms, a technique for quantifying and reducing uncertainty in vision based object detection through object specific transformations at both training and inference times. At training time, ObjectTransforms perform color space perturbations on individual objects, improving robustness to lighting and color variations. ObjectTransforms also uses diffusion models to generate realistic, diverse pedestrian instances. At inference time, object perturbations are applied to detected objects and the variance of detection scores are used to quantify predictive uncertainty in real time. This uncertainty signal is then used to filter out false positives and also recover false negatives, improving the overall precision recall curve. Experiments with YOLOv8 on the NuImages 10K dataset demonstrate that our method yields notable accuracy improvements and uncertainty reduction across all object classes during training, while predicting desirably higher uncertainty values for false positives as compared to true positives during inference. Our results highlight the potential of ObjectTransforms as a lightweight yet effective mechanism for reducing and quantifying uncertainty in vision-based perception during training and inference respectively.

Figures (5)

• Figure 1: Night-time crosswalk event captured from a real AV. This is out of distribution data not used during training and validation of the YOLOv8 model. [Left] baseline detector at conf = 0.1 misses a pedestrian and a car. [Middle] Lowering to conf = 0.05 recovers these but introduces false positives. [Right] Our ObjectTransforms-based uncertainty filtering suppresses the false positives while retaining the true positives.
• Figure 2: The diffusion-based pedestrian augmentation pipeline.
• Figure 3: The Object Consistency Filter (OCF) using a CLIP encoder filters out incorrect outputs from diffusion-model inpainting
• Figure 4: MC Dropout Uncertainty on NuImages nuscenes Test Dataset (10 passes, Conf=0.5).
• Figure 5: Distribution of TP and FP along the uncertainty score. A threshold of U = 0.19 for conf = 0.01 (refer to Table \ref{['tab:tp_fp']})