VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection

TL;DR

Open-vocabulary aerial object detection requires recognizing novel categories beyond predefined labels, but prior approaches rely on text supervision and suffer from localization and semantic-noise issues. VK-Det introduces Visual Knowledge Guided Prototype Learning, combining Adaptive Selection Knowledge Distillation (ASKD) to exploit informative regions, Prototype-Aware Pseudo-Labeling (PAPL) to form latent unknown-category prototypes via clustering, and Synthetic Matching Inference (SMI) to fuse distillation, prototype scores, and localization signals. The method achieves state-of-the-art performance on DIOR ($mAP^{N}$ = 30.1%) and DOTA ($mAP^{N}$ = 23.3%), outperforming even methods with extra supervision, and demonstrates that visual knowledge from VLMs can drive effective open-vocabulary detection in aerial imagery. This work highlights a practical, data-efficient path for OVAD in dense, real-world settings and suggests future directions for lightweight, scalable deployment of VLM-guided dense prediction.

Abstract

To identify objects beyond predefined categories, open-vocabulary aerial object detection (OVAD) leverages the zero-shot capabilities of visual-language models (VLMs) to generalize from base to novel categories. Existing approaches typically utilize self-learning mechanisms with weak text supervision to generate region-level pseudo-labels to align detectors with VLMs semantic spaces. However, text dependence induces semantic bias, restricting open-vocabulary expansion to text-specified concepts. We propose $\textbf{VK-Det}$, a $\textbf{V}$isual $\textbf{K}$nowledge-guided open-vocabulary object $\textbf{Det}$ection framework $\textit{without}$ extra supervision. First, we discover and leverage vision encoder's inherent informative region perception to attain fine-grained localization and adaptive distillation. Second, we introduce a novel prototype-aware pseudo-labeling strategy. It models inter-class decision boundaries through feature clustering and maps detection regions to latent categories via prototype matching. This enhances attention to novel objects while compensating for missing supervision. Extensive experiments show state-of-the-art performance, achieving 30.1 $\mathrm{mAP}^{N}$ on DIOR and 23.3 $\mathrm{mAP}^{N}$ on DOTA, outperforming even extra supervised methods.

Figures (8)

• Figure 1: (a) visualizes the attention heatmap from the visual encoder of VLMs for an aerial image. The heatmap is derived by averaging multi-layer attentions. (b) compares our pseudo-labeling approach with conventional methods.
• Figure 2: The overall architecture of our VK-Det. By utilizing ASKD and PAPL in training, the detector enables comprehensive learning of unlabeled objects. During inference, SMI systematically evaluate novel category objects. In the figure, RMCLIP denotes RemoteCLIP7.
• Figure 3: Visualization of open-vocabulary inference on DIOR dataset
• Figure 4: Comparison of feature distributions: (a) Feature distribution in pseudo-labeled data with high IoU to the ground truth labels of novel classes; (b) Feature distribution in pseudo-labeled data exhibiting high similarity to the text embeddings of novel classes.
• Figure 5: Workflow of how to classify the pseudo-labeled data.