DVGBench: Implicit-to-Explicit Visual Grounding Benchmark in UAV Imagery with Large Vision-Language Models

TL;DR

This work introduces DVGBench, a UAV-focused implicit visual grounding benchmark with explicit-implicit query pairs to assess reasoning in remote-sensing LVLMs. It será a two-stage DroneVG-R1 model combining GRPO-based reinforcement fine-tuning with an I2E-CoT inference strategy to convert implicit references into explicit groundings, enabling improved region-level and pixel-level grounding. Extensive experiments show state-of-the-art performance for DroneVG-R1, especially when paired with strong segmentation backbones, and reveal the critical role of reasoning-based rewards and explicit textual cues in grounding accuracy. The results highlight the importance of explicit reasoning pathways for implicit VG in drone contexts and establish a foundation for future improvements in UAV agents’ reasoning capabilities.

Abstract

Remote sensing (RS) large vision-language models (LVLMs) have shown strong promise across visual grounding (VG) tasks. However, existing RS VG datasets predominantly rely on explicit referring expressions-such as relative position, relative size, and color cues-thereby constraining performance on implicit VG tasks that require scenario-specific domain knowledge. This article introduces DVGBench, a high-quality implicit VG benchmark for drones, covering six major application scenarios: traffic, disaster, security, sport, social activity, and productive activity. Each object provides both explicit and implicit queries. Based on the dataset, we design DroneVG-R1, an LVLM that integrates the novel Implicit-to-Explicit Chain-of-Thought (I2E-CoT) within a reinforcement learning paradigm. This enables the model to take advantage of scene-specific expertise, converting implicit references into explicit ones and thus reducing grounding difficulty. Finally, an evaluation of mainstream models on both explicit and implicit VG tasks reveals substantial limitations in their reasoning capabilities. These findings provide actionable insights for advancing the reasoning capacity of LVLMs for drone-based agents. The code and datasets will be released at https://github.com/zytx121/DVGBench

Figures (17)

• Figure 1: Infants can understand references composed of colors and relative positions easily, but cannot comprehend references involving common sense or domain knowledge. We refer to the latter as Implicit Visual Grounding.
• Figure 2: Overview of the Implicit-to-Explicit mechanism. This diagram compares the standard Group Relative Policy Optimization (GRPO) with our I2E-CoT approach. The GRPO mislocates the left-turning vehicle due to visual attention distraction during reasoning. In contrast, the I2E-CoT method employs the <explicit> token to generate an explicit reference for the object, correcting the initial localization and producing the correct answer. Attention graphs reveal that during the <explicit> phase, I2E-CoT identifies the explicit "green" cue, substantially increasing attention to the corresponding image tokens (blue line).
• Figure 3: Visualization of the six UAV application scenarios in DVGBench. Each of the main scenarios also includes some sub-scenes. It is worth mentioning that all questions are manually labeled, rather than generated by LVLMs. Therefore, the questions cover a diverse range of knowledge points and are more challenging compared to existing RS VG datasets.
• Figure 4: Framework of the DroneVG-R1, which comprises a reasoning model and a segmentation model. The reasoning model is an LVLM that generates reasoning chains and provides box-level results. Subsequently, the segmentation model produces a pixel-wise mask based on the box. In addition to regular format rewards and perceptual rewards, we have also designed a reasoning reward to enhance the quality of the model's implicit-to-explicit conversion through human-annotated explicit references.
• Figure 5: Prompt templates of CoT and I2E-CoT.