Leveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging

TL;DR

This work tackles confidence calibration for microscopy object detectors by exploiting multi-rater annotations to model aleatoric uncertainty. The authors train separate detectors for each rater and ensemble their predictions, enabling the ensemble to reflect annotator disagreement and reduce overconfidence. On a colorectal organoid brightfield dataset, the rater-specific ensemble achieves substantially better calibration (lower $D{-}ECE$) than a baseline mixed-label ensemble while preserving detection accuracy, illustrating the practical value of explicitly modeling inter-rater biases. The approach highlights that accounting for annotator variability can yield more trustworthy detectors for biomedical imaging, with clear trade-offs in computational cost and data requirements.

Abstract

Deep learning-based object detectors have achieved impressive performance in microscopy imaging, yet their confidence estimates often lack calibration, limiting their reliability for biomedical applications. In this work, we introduce a new approach to improve model calibration by leveraging multi-rater annotations. We propose to train separate models on the annotations from single experts and aggregate their predictions to emulate consensus. This improves upon label sampling strategies, where models are trained on mixed annotations, and offers a more principled way to capture inter-rater variability. Experiments on a colorectal organoid dataset annotated by two experts demonstrate that our rater-specific ensemble strategy improves calibration performance while maintaining comparable detection accuracy. These findings suggest that explicitly modelling rater disagreement can lead to more trustworthy object detectors in biomedical imaging.

Figures (4)

• Figure 1: Multi-rater annotations of healthy organoids. Shown are merged detections where raters annotations matched ($\text{IoU} \ge 0.5$), and detections that were annotated by only one rater.
• Figure 2: Inter-rater statistics on the single-rater dataset. (a) Mean number of objects per image retained from the pre-trained model, manually added by each rater, and total number of annotated objects. (b) F1-score between the two raters’ annotations across varying IoU thresholds averaged over all images. Error bars indicate standard deviation.
• Figure 3: Calibration plots for the two ensembling strategies with $20$ models. Blue bars show the average precision observed in each bin $B_i$ computed at thresholds of $0.5$ and $0.75$, while red bars indicate the gap to a perfectly calibrated model. The is reported for $=0.5$.
• Figure 4: and of the ensemble models for different ensemble sizes. Results are averaged over 100 bootstrap iterations; error bars represent standard deviation.