Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation

TL;DR

This paper addresses whether open-vocabulary object detectors trained on natural, ground-level imagery can transfer to aerial imagery without fine-tuning. It benchmarks five state-of-the-art OVD models on the LAE-80C aerial dataset under strict zero-shot regimes, using Global, Oracle, and Single-Category inference to separate semantic confusion from localization. The findings reveal a severe domain transfer gap, with the best model achieving $F1=0.276$ and a $69\%$ false-positive rate, and show that reducing vocabulary size (to ~3.2 classes) yields ~15× improvement, highlighting semantic confusion as the primary bottleneck. Prompt engineering strategies provide little benefit, and performance varies drastically across datasets, underscoring the need for domain-adaptive pretraining and bespoke evaluation protocols for aerial OVD applications.

Abstract

Open-vocabulary object detection (OVD) enables zero-shot recognition of novel categories through vision-language models, achieving strong performance on natural images. However, transferability to aerial imagery remains unexplored. We present the first systematic benchmark evaluating five state-of-the-art OVD models on the LAE-80C aerial dataset (3,592 images, 80 categories) under strict zero-shot conditions. Our experimental protocol isolates semantic confusion from visual localization through Global, Oracle, and Single-Category inference modes. Results reveal severe domain transfer failure: the best model (OWLv2) achieves only 27.6% F1-score with 69% false positive rate. Critically, reducing vocabulary size from 80 to 3.2 classes yields 15x improvement, demonstrating that semantic confusion is the primary bottleneck. Prompt engineering strategies such as domain-specific prefixing and synonym expansion, fail to provide meaningful performance gains. Performance varies dramatically across datasets (F1: 0.53 on DIOR, 0.12 on FAIR1M), exposing brittleness to imaging conditions. These findings establish baseline expectations and highlight the need for domain-adaptive approaches in aerial OVD.

Figures (7)

• Figure 1: Detection pipeline for open-vocabulary aerial object detection. The pipeline progressively filters detections using box confidence, text alignment, class-wise NMS, and final score thresholds.
• Figure 2: Comparison of Precision, Recall, and F1 Score for all evaluated models.
• Figure 3: True Positive (TP), False Positive (FP), and False Negative (FN) counts across models.
• Figure 4: Precision-Recall Trade-off. DINO variants optimize for precision at the expense of coverage, while OWLv2 sacrifices precision for higher recall. Ideally, a model would appear in the top-right corner.
• Figure 5: Conservative detection behavior. Grounding DINO collapses multiple nearby instances into a single high-confidence detection, prioritizing precision over instance-level recall.