MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection

TL;DR

MoCaE addresses the miscalibration barrier in mixing object detectors by calibrating each expert’s confidences and aggregating via Refining NMS (Soft NMS with Score Voting). The approach yields consistent gains across diverse tasks (COCO, LVIS, DOTA, OVOD) and sets state-of-the-art performance on COCO test-dev and DOTA, while remaining simple to apply to off-the-shelf detectors. Theoretical and empirical analyses show calibration is the primary driver of gains, with Oracle calibrators representing upper bounds. The work demonstrates that principled, calibration-aware ensembles can reliably leverage detector diversity for improved detection and segmentation, with practical impact for real-world detection systems. It also highlights limitations when detector performance gaps are large and emphasizes continued research in calibration methods and resource-aware deployment.

Abstract

Combining the strengths of many existing predictors to obtain a Mixture of Experts which is superior to its individual components is an effective way to improve the performance without having to develop new architectures or train a model from scratch. However, surprisingly, we find that naïvely combining expert object detectors in a similar way to Deep Ensembles, can often lead to degraded performance. We identify that the primary cause of this issue is that the predictions of the experts do not match their performance, a term referred to as miscalibration. Consequently, the most confident detector dominates the final predictions, preventing the mixture from leveraging all the predictions from the experts appropriately. To address this, when constructing the Mixture of Experts, we propose to combine their predictions in a manner which reflects the individual performance of the experts; an objective we achieve by first calibrating the predictions before filtering and refining them. We term this approach the Mixture of Calibrated Experts and demonstrate its effectiveness through extensive experiments on 5 different detection tasks using a variety of detectors, showing that it: (i) improves object detectors on COCO and instance segmentation methods on LVIS by up to $\sim 2.5$ AP; (ii) reaches state-of-the-art on COCO test-dev with $65.1$ AP and on DOTA with $82.62$ $\mathrm{AP_{50}}$; (iii) outperforms single models consistently on recent detection tasks such as Open Vocabulary Object Detection.

Figures (9)

• Figure 1: Piecharts showing % of detections from three similarly performing detectors in their resulting s on COCO dataset. (a) of uncalibrated detectors, (b) of calibrated detectors, and (c) histogram of confidence scores.
• Figure 2: Detections are color-coded. red: RS R-CNN, blue: ATSS, green: PAA. (a-c) Outputs of the detectors on an example image. RS R-CNN misses the "surfboard", ATSS misses a "person", PAA has a notable localisation error for the "person" in front seat. (d-f) The detections from of uncalibrated detectors; of calibrated detectors; and the ground truth. (d) is dominated by the most confident RS R-CNN and misses the "surfboard". After calibration in (e), all objects are detected accurately by improving each expert.
• Figure 3: Reliability diagrams calibration of (a) RS R-CNN and (b) ATSS.
• Figure 4: pipeline. Given an image $X$, each detector follows its own pipeline including postprocessing (in orange) and outputs $\{\hat{p},\hat{b}\}$. Without any modification to the pipeline of each detector, we introduce two modules in blue: (i) calibrate the confidence scores of each detector; (ii) aggregate them via Refining providing the detections of $\{\hat{p}^{MoE},\hat{b}^{MoE}\}$.
• Figure A.5: Early calibration. Raw confidence scores $\hat{p}^{raw}$ are calibrated, and the standard post-processing steps handle aggregation in which (in blue) removes the duplicates from multiple detectors.

Theorems & Definitions (4)

• Lemma 1
• proof
• Theorem 1
• proof