Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation

TL;DR

This work defines open-vocabulary aerial object detection (OVAD) and introduces CastDet, a CLIP-activated student–teacher detector that operates on horizontal and oriented bounding boxes in aerial imagery. CastDet combines a continuously updated localization teacher (EMA of the student) with an external vision–language teacher (RemoteCLIP) and a dynamic pseudo-label queue to progressively expand the detectable vocabulary using unlabeled data. It introduces horizontal and oriented box representations, plus a suite of box-selection strategies (RPN, RJV, BJV, SJV, AJV) and a hybrid training objective that balances supervision, unlabeled guidance, and queue-derived signals via $\mathcal{L}=\alpha\mathcal{L}_s+\beta\mathcal{L}_u+\gamma\mathcal{L}_d$. Extensive experiments on DIOR, DOTA, STAR, VisDroneZSD, and related aerial benchmarks show significant improvements over baselines and existing SOTA in open-vocabulary performance, and demonstrate the value of a dynamic labeling mechanism for continual vocabulary expansion in aerial scenes.

Abstract

In recent years, aerial object detection has been increasingly pivotal in various earth observation applications. However, current algorithms are limited to detecting a set of pre-defined object categories, demanding sufficient annotated training samples, and fail to detect novel object categories. In this paper, we put forth a novel formulation of the aerial object detection problem, namely open-vocabulary aerial object detection (OVAD), which can detect objects beyond training categories without costly collecting new labeled data. We propose CastDet, a CLIP-activated student-teacher detection framework that serves as the first OVAD detector specifically designed for the challenging aerial scenario, where objects often exhibit weak appearance features and arbitrary orientations. Our framework integrates a robust localization teacher along with several box selection strategies to generate high-quality proposals for novel objects. Additionally, the RemoteCLIP model is adopted as an omniscient teacher, which provides rich knowledge to enhance classification capabilities for novel categories. A dynamic label queue is devised to maintain high-quality pseudo-labels during training. By doing so, the proposed CastDet boosts not only novel object proposals but also classification. Furthermore, we extend our approach from horizontal OVAD to oriented OVAD with tailored algorithm designs to effectively manage bounding box representation and pseudo-label generation. Extensive experiments for both tasks on multiple existing aerial object detection datasets demonstrate the effectiveness of our approach. The code is available at https://github.com/VisionXLab/CastDet.

Figures (14)

• Figure 1: Comparisons of the amount of categories and images of 19 common aerial and natural image datasets. Aerial datasets are much smaller in size and category vocabularies compared with natural image datasets.
• Figure 2: Class-agnostic RPN recall statistics of novel categories in the natural dataset COCO lin2014microsoft and the aerial dataset VisDroneZSD VisDrone2023 (i.e., 77% v.s. 48%). The recall of novel objects in aerial images is much lower than that in natural images due to the highly complex backgrounds.
• Figure 3: Aerial images from DIOR dior_li2020object. (a)(b) Objects in aerial images exhibit background interference. The red box represents the ground truth, while the yellow box highlights the areas of interference. (c) Inaccurate localization with horizontal bounding box (HBox), causing significant background clutter in aerial images compared to oriented bounding box (RBox).
• Figure 4: Overall architecture of CastDet. In each training iteration, the data batch consists of three data flow: labeled data with annotations, unlabeled data, and data sampled from the dynamic label queue. The labeled images are directly used for the student model training (rgb]0.84,0.894,0.805$\mathcal{L}_s$), while two sets of pseudo-labels of unlabeled data are predicted through the localization teacher model and external teacher model. One supervises the student model (rgb]0.726,0.816,0.898$\mathcal{L}_u$), and the other is pushed into the dynamic label queue. Simultaneously, samples are randomly selected from the dynamic label queue to enhance the student's ability to detect novel objects (rgb]0.99,0.94,0.805$\mathcal{L}_d$).
• Figure 5: Augmentation strategies used in our approach.Weak augmentation applies only random flip (highlighted in orange box), whereas strong augmentation incorporates all augmentation techniques.