Training-free Boost for Open-Vocabulary Object Detection with Confidence Aggregation

TL;DR

This work tackles open-vocabulary object detection (OVOD) where novel classes are undervalued by detector confidence and suppressed by NMS due to training bias in objectness and region-text similarity. It proposes AggDet, a training-free post-processing framework that aggregates confidence from two stages: region-proposal localization quality and text-guided visual prototype similarities, including DeltaEdit-based prototype extrapolation for novel categories. Key formulas define the aggregation: q_i = (1/K) sum_{j in TopK(P_{i,:})} P_{i,j} and o_i^{Agg} = 0.5*(o_i + q_i); for classification, s^{Agg}_{i,k} = f_i · t_k + alpha f_i · p_k with c^{Agg}_{i,k} = sigma(s^{Agg}_{i,k}), plus prototype extrapolation hat{p}_k - p_bar ≈ t_k - t_bar and tilde{c}_{i,*}^{Agg} = (c^{Agg}_{i,*})^{gamma} (q_i)^{1-gamma}. Experiments on OV-COCO and OV-LVIS show consistent gains across detectors (up to +3.3% on OV-COCO and +1.5% on OV-LVIS) with negligible latency, demonstrating that confidence aggregation at inference is a practical, model-agnostic boost for OVOD that can inform future training-time enhancements.

Abstract

Open-vocabulary object detection (OVOD) aims at localizing and recognizing visual objects from novel classes unseen at the training time. Whereas, empirical studies reveal that advanced detectors generally assign lower scores to those novel instances, which are inadvertently suppressed during inference by commonly adopted greedy strategies like Non-Maximum Suppression (NMS), leading to sub-optimal detection performance for novel classes. This paper systematically investigates this problem with the commonly-adopted two-stage OVOD paradigm. Specifically, in the region-proposal stage, proposals that contain novel instances showcase lower objectness scores, since they are treated as background proposals during the training phase. Meanwhile, in the object-classification stage, novel objects share lower region-text similarities (i.e., classification scores) due to the biased visual-language alignment by seen training samples. To alleviate this problem, this paper introduces two advanced measures to adjust confidence scores and conserve erroneously dismissed objects: (1) a class-agnostic localization quality estimate via overlap degree of region/object proposals, and (2) a text-guided visual similarity estimate with proxy prototypes for novel classes. Integrated with adjusting techniques specifically designed for the region-proposal and object-classification stages, this paper derives the aggregated confidence estimate for the open-vocabulary object detection paradigm (AggDet). Our AggDet is a generic and training-free post-processing scheme, which consistently bolsters open-vocabulary detectors across model scales and architecture designs. For instance, AggDet receives 3.3% and 1.5% gains on OV-COCO and OV-LVIS benchmarks respectively, without any training cost.

Figures (9)

• Figure 1: Left: the region-proposal containing novel objects (colored in pink) is suppressed due to its lower objectness score (0.15). Right: detectors generally assign lower classification scores for base classes than novel classes.
• Figure 2: The framework of our proposed AggDet, which aggregates the confidence estimates at both region-proposal and object-classification stages.
• Figure 3: An illustration depicting proposals clustered around the ground truth objects.
• Figure 4: Left: Our aggregated confidence score successfully preserved the validated proposals (colored in green). Right: the recall difference for novel classes between the baseline method and our proposed AggDet.
• Figure 5: Extrapolation for visual prototypes on novel categories.