Beyond Open Vocabulary: Multimodal Prompting for Object Detection in Remote Sensing Images

TL;DR

Open-vocabulary object detection in remote sensing is challenged by domain-specific category semantics that impair text-only grounding. RS-MPOD addresses this by introducing an instance-grounded visual prompt encoder and a multimodal prompt fusion module built on a GroundingDINO backbone, enabling category specification via visual exemplars, text, or their combination and training in a staged manner. Across in-domain, cross-dataset, and fine-grained RS benchmarks, visual prompting enhances robustness to semantic ambiguity and distribution shifts, while multimodal prompting offers additional gains when textual semantics align with the task. This framework advances practical open-vocabulary RS detection by enabling flexible, appearance-based category specification that complements textual semantics, improving reliability in diverse remote sensing applications.

Abstract

Open-vocabulary object detection in remote sensing commonly relies on text-only prompting to specify target categories, implicitly assuming that inference-time category queries can be reliably grounded through pretraining-induced text-visual alignment. In practice, this assumption often breaks down in remote sensing scenarios due to task- and application-specific category semantics, resulting in unstable category specification under open-vocabulary settings. To address this limitation, we propose RS-MPOD, a multimodal open-vocabulary detection framework that reformulates category specification beyond text-only prompting by incorporating instance-grounded visual prompts, textual prompts, and their multimodal integration. RS-MPOD introduces a visual prompt encoder to extract appearance-based category cues from exemplar instances, enabling text-free category specification, and a multimodal fusion module to integrate visual and textual information when both modalities are available. Extensive experiments on standard, cross-dataset, and fine-grained remote sensing benchmarks show that visual prompting yields more reliable category specification under semantic ambiguity and distribution shifts, while multimodal prompting provides a flexible alternative that remains competitive when textual semantics are well aligned.

Figures (4)

• Figure 1: Prompting paradigms for object detection. (a) Open-vocabulary prompting only supports textual prompts. (b) Open prompt extends prompts to textual and visual inputs via an external pre-trained prompt encoder. (c) Our multimodal prompting jointly supports textual, visual, and multimodal prompts with trainable prompt encoders.
• Figure 2: Overall framework of RS-MPOD. The detector is built upon GroundingDINO and supports textual prompting, visual prompting, and multimodal prompting within a unified detection pipeline. Prompt embeddings produced by different prompt encoders are used to condition query selection and cross-attention in the transformer decoder for category specification. The lower panels illustrate the designs of the textual prompt encoder, visual prompt encoder, and the multimodal prompt fusion module.
• Figure 3: Stage-wise training strategy of RS-MPOD. The detector is first trained with textual prompts, followed by visual prompt encoder training and multimodal prompt fusion, with earlier components frozen in later stages. The image encoder consists of an image backbone and a transformer encoder.
• Figure 4: Qualitative comparison of detection results under different prompting strategies. From left to right: ground truth, text-only prompting, visual-only prompting, and multimodal prompting. The three rows illustrate representative scenarios with varying degrees of semantic alignment between pretraining and evaluation: (top) ship detection with category name mismatch (boat vs. ship), (middle) car detection with weak textual representation and confusion with visually similar categories (e.g., van), and (bottom) helicopter detection with strong semantic alignment in pretraining. Multimodal prompting exhibits more balanced behavior across scenarios by combining complementary textual and visual cues.