Do Object Detection Localization Errors Affect Human Performance and Trust?

TL;DR

This paper investigates whether bounding box localization errors affect human performance and trust when humans use automated object detections. Using an observer-performance paradigm on a visual multi-object counting task, it manipulates localization quality via $IoU$-based shifts and $F1$-score perturbations, and compares standard bounding boxes to center-dot visualizations. The authors find that localization errors in position alone do not significantly impair accuracy or trust, while errors in detection count (false positives/negatives) degrade both performance and trust; center dots can substantially improve counting accuracy and robustness to localization inaccuracies. Practically, the work suggests optimizing $F1$-score rather than IoU for human-in-the-loop systems and demonstrates simple visualization choices can meaningfully boost user performance and resilience to localization errors.

Abstract

Bounding boxes are often used to communicate automatic object detection results to humans, aiding humans in a multitude of tasks. We investigate the relationship between bounding box localization errors and human task performance. We use observer performance studies on a visual multi-object counting task to measure both human trust and performance with different levels of bounding box accuracy. The results show that localization errors have no significant impact on human accuracy or trust in the system. Recall and precision errors impact both human performance and trust, suggesting that optimizing algorithms based on the F1 score is more beneficial in human-computer tasks. Lastly, the paper offers an improvement on bounding boxes in multi-object counting tasks with center dots, showing improved performance and better resilience to localization inaccuracy.

Figures (4)

• Figure 1: Illustration visualizing improvement of human performance in human computer task. A design choice focused on the human in the system could improve performance without need of object detector improvement.
• Figure 2: Example of images used within the study. Going from left to right accurate bounding boxes (Perfect box), bounding boxes with 0.5 IoU (Shifted box), accurate Dots (Perfect Dot ), Inaccurate dots (Shifted Dot), additional boxes(False positive) and missing boxes(False negative).
• Figure 3: Boxplot comparing the absolute error of the different experiments.
• Figure 4: Boxplot comparing response times within the different experiments.