TCMA: Text-Conditioned Multi-granularity Alignment for Drone Cross-Modal Text-Video Retrieval

TL;DR

This work tackles drone video–text retrieval by introducing the DVTMD dataset with fine-grained, diverse captions and proposing TCMA, a multi-granularity alignment framework that fuses global semantics with sentence-guided frame weighting and word-guided patch attention. The model leverages a CLIP-based pretraining backbone, a three-stage aggregation pipeline, and a hierarchical contrastive objective with Pearson regularization to capture both global context and local details. Extensive experiments on DVTMD and CapERA show state-of-the-art performance, with substantial improvements in R@1 for text-to-video and video-to-text retrieval under different backbones, demonstrating the value of fine-grained UAV annotations and adaptive, text-conditioned attention. The work provides a practical, scalable approach for robust drone content retrieval and offers a valuable benchmark and codebase for the community.

Abstract

Unmanned aerial vehicles (UAVs) have become powerful platforms for real-time, high-resolution data collection, producing massive volumes of aerial videos. Efficient retrieval of relevant content from these videos is crucial for applications in urban management, emergency response, security, and disaster relief. While text-video retrieval has advanced in natural video domains, the UAV domain remains underexplored due to limitations in existing datasets, such as coarse and redundant captions. Thus, in this work, we construct the Drone Video-Text Match Dataset (DVTMD), which contains 2,864 videos and 14,320 fine-grained, semantically diverse captions. The annotations capture multiple complementary aspects, including human actions, objects, background settings, environmental conditions, and visual style, thereby enhancing text-video correspondence and reducing redundancy. Building on this dataset, we propose the Text-Conditioned Multi-granularity Alignment (TCMA) framework, which integrates global video-sentence alignment, sentence-guided frame aggregation, and word-guided patch alignment. To further refine local alignment, we design a Word and Patch Selection module that filters irrelevant content, as well as a Text-Adaptive Dynamic Temperature Mechanism that adapts attention sharpness to text type. Extensive experiments on DVTMD and CapERA establish the first complete benchmark for drone text-video retrieval. Our TCMA achieves state-of-the-art performance, including 45.5% R@1 in text-to-video and 42.8% R@1 in video-to-text retrieval, demonstrating the effectiveness of our dataset and method. The code and dataset will be released.

Figures (9)

• Figure 1: Illustration of the characteristic in the drone text-video retrieval task. Video-level: global sentence and video encompass the overall semantics. Frame-level: due to the rapid movement and wide field of view of the drone, there are many frames with low relevance to the query text that need to be filtered. Patch-level: Since aerial video frames often contain extensive background regions and irrelevant objects, only a limited part of the frame content is semantically relevant to the words in the query text.
• Figure 2: The construction pipeline of our DVTMD benchmark: Step 1) frame sampling, Step 2) detailed frame description generation, Step 3) video-level summarization, and Step 4) dataset formation.
• Figure 3: The designed prompt template for generating detailed and objective frame-level descriptions.
• Figure 4: The designed prompt template for generating summarized captions.
• Figure 5: Graphical analysis of the captions. (a) Caption length distribution across videos, with most captions falling between 15 and 25 words, peaking around 18–22 words. (b) Number of captions per category in the training and testing splits. The "NonEvent" category exhibits a minor discrepancy between the splits while other categories maintain a balanced distribution. (c),(d) Word clouds of the train set and test set captions displaying various elements, including humans, actions, objects, and backgrounds.