Aligning Object Detector Bounding Boxes with Human Preference

TL;DR

This work investigates how to align automatically detected object boxes with human preference and investigates whether this improves human quality perception and proposes an asymmetric bounding box regression loss that encourages large over small predicted bounding boxes.

Abstract

Previous work shows that humans tend to prefer large bounding boxes over small bounding boxes with the same IoU. However, we show here that commonly used object detectors predict large and small boxes equally often. In this work, we investigate how to align automatically detected object boxes with human preference and study whether this improves human quality perception. We evaluate the performance of three commonly used object detectors through a user study (N = 123). We find that humans prefer object detections that are upscaled with factors of 1.5 or 2, even if the corresponding AP is close to 0. Motivated by this result, we propose an asymmetric bounding box regression loss that encourages large over small predicted bounding boxes. Our evaluation study shows that object detectors fine-tuned with the asymmetric loss are better aligned with human preference and are preferred over fixed scaling factors. A qualitative evaluation shows that human preference might be influenced by some object characteristics, like object shape.

Figures (7)

• Figure 1: Scaling the predicted bounding box of Faster R-CNN ren2015faster on the COCO lin2014microsoft validation set. Average Precision (AP) (top) versus human preference (bottom). A scaling factor of $1.0$ corresponds to the original bounding box size. Upscaling and downscaling the size of the bounding boxes severely deteriorates AP. However, our study shows that humans prefer larger bounding boxes, even if they give nearly 0 AP.
• Figure 2: Amount of large and small bounding boxes are predicted by three object detectors on the MS COCO dataset, for seven IoU intervals, ranging from 0.3 to 1.0. For all three detectors, with higher IoU thresholds more small than large boxes are detected.
• Figure 3: Scaling the model detections. Example of a bounding box predicted for a large object (first row), a medium object (second row) and a small object (third row) with Faster R-CNN (3rd column) and its scaled versions. In the left two images, the area of the bounding box is reduced by a scaling factor of, respectively, $0.5$ and $0.67$, whilst in the right two images the box area is increased by a factor of $1.5$ and $2$.
• Figure 4: Results from the Scaling Preference user study. The histograms show the percentage of preferred bounding box size per object category (S, M, L) and IoU range, from $0.5 \leq \text{IoU} < 0.6$ to $0.9\leq \text{IoU} < 1.0$, for three object detectors. The plots indicate that humans significantly prefer larger boxes.
• Figure 5: Asymmetric smooth $L_1$ loss with different $\alpha$. Larger bounding boxes are penalized less than smaller predicted boxes.