Towards Robust Cross-Dataset Object Detection Generalization under Domain Specificity

TL;DR

This work introduces setting specificity as a principled lens to study cross-dataset object detection, distinguishing setting-specific from setting-agnostic benchmarks and analyzing transfer under a fixed training protocol without target access. By evaluating a standard detector across four datasets and employing both closed-label and open-label (CLIP-guided) assessments, it disentangles visual domain shift from label taxonomy mismatch and reveals robust within-setting transfer but substantial cross-setting fragility, including asymmetric transfers. The study provides semantic near-miss diagnostics to characterize residual failures and demonstrates that open-label gains are meaningful yet bounded, underscoring the primacy of distribution shift over taxonomy changes in hardest regimes. The findings offer practical guidance for evaluating detectors under distribution shift and motivate future work on domain-generalization and setting-aware adaptation for safety-critical deployments, with code to be released publicly.

Abstract

Object detectors often perform well in-distribution, yet degrade sharply on a different benchmark. We study cross-dataset object detection (CD-OD) through a lens of setting specificity. We group benchmarks into setting-agnostic datasets with diverse everyday scenes and setting-specific datasets tied to a narrow environment, and evaluate a standard detector family across all train--test pairs. This reveals a clear structure in CD-OD: transfer within the same setting type is relatively stable, while transfer across setting types drops substantially and is often asymmetric. The most severe breakdowns occur when transferring from specific sources to agnostic targets, and persist after open-label alignment, indicating that domain shift dominates in the hardest regimes. To disentangle domain shift from label mismatch, we compare closed-label transfer with an open-label protocol that maps predicted classes to the nearest target label using CLIP similarity. Open-label evaluation yields consistent but bounded gains, and many corrected cases correspond to semantic near-misses supported by the image evidence. Overall, we provide a principled characterization of CD-OD under setting specificity and practical guidance for evaluating detectors under distribution shift. Code will be released at \href{[https://github.com/Ritabrata04/cdod-icpr.git}{https://github.com/Ritabrata04/cdod-icpr}.

Figures (4)

• Figure 1: Object detector inference on an image from the COCO dataset. (a) shows the In-domain performance of COCO, whereas (b), (c), and (d) show the Cross-domain performance for a model trained on Cityscapes, Objects365, and BDD100k respectively.
• Figure 2: Setting-specific examples across different domains. (a) and (b) are BDD100k images, whereas (c) and (d) are Cityscapes images
• Figure 3: Setting-agnostic examples. (a) and (b) are COCO images whereas (c) and (d) are Objects365 images
• Figure 4: In-domain and Cross-domain evaluation results. City refers to images from Cityscapes, and O365 refers to images from the Objects365 dataset. X$\rightarrow$Y refers to the inference setting of the model being trained on the dataset X and tested on the dataset Y