MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval

TL;DR

This work tackles drone video-text retrieval by introducing MSAM, which combines Multi-Semantic Adaptive Learning Mechanism (MSALM) with a Cross-Modal Interactive Feature Fusion Pooling (CIFFP) to enable fine-grained, region-aware cross-modal matching. The model encodes text and video with CLIP-based backbones into a shared latent space of dimension $D$ and generates $k$ probabilistic embeddings per modality to realize one-to-many semantic alignment. The training objective fuses a video-text matching term with distribution-driven and diversity constraints, enabling robust cross-modal reasoning in visually homogeneous drone scenes. Two drone-specific datasets USRD and UMCRD demonstrate state-of-the-art results and illustrate the practical value of region-focused, multi-semantic retrieval for drone analytics and surveillance tasks.

Abstract

With the advancement of drone technology, the volume of video data increases rapidly, creating an urgent need for efficient semantic retrieval. We are the first to systematically propose and study the drone video-text retrieval (DVTR) task. Drone videos feature overhead perspectives, strong structural homogeneity, and diverse semantic expressions of target combinations, which challenge existing cross-modal methods designed for ground-level views in effectively modeling their characteristics. Therefore, dedicated retrieval mechanisms tailored for drone scenarios are necessary. To address this issue, we propose a novel approach called Multi-Semantic Adaptive Mining (MSAM). MSAM introduces a multi-semantic adaptive learning mechanism, which incorporates dynamic changes between frames and extracts rich semantic information from specific scene regions, thereby enhancing the deep understanding and reasoning of drone video content. This method relies on fine-grained interactions between words and drone video frames, integrating an adaptive semantic construction module, a distribution-driven semantic learning term and a diversity semantic term to deepen the interaction between text and drone video modalities and improve the robustness of feature representation. To reduce the interference of complex backgrounds in drone videos, we introduce a cross-modal interactive feature fusion pooling mechanism that focuses on feature extraction and matching in target regions, minimizing noise effects. Extensive experiments on two self-constructed drone video-text datasets show that MSAM outperforms other existing methods in the drone video-text retrieval task. The source code and dataset will be made publicly available.

Figures (8)

• Figure 1: Top row: Comparism of common video-text matching and drone video-text matching. Bottom row: The video frames from the same scene in the drone dataset illustrate the strong structural homogeneity of drone videos, while the target combinations within a single video exhibit semantic diversity in their descriptions.
• Figure 2: The overall overview of the MSAM framework is as follows. For a given drone video-text pair $P = {\{\boldsymbol{v}, \boldsymbol{t}\}}$, we first encode it using the text encoder $\mathcal{C}_t$ to obtain $\boldsymbol{N_t}$ and $\boldsymbol{f_t}$. Similarly, the image encoder $\mathcal{C}_v$ is used to encode the frames of the given video $\boldsymbol{v}$, generating $\boldsymbol{N_v}$ and $\boldsymbol{f_v}$. Then the similarity between $\boldsymbol{N_v}$ and $\boldsymbol{f_t}$ is computed by Cross-Modal Interactive Feature Fusion Pooling and optimize through $\mathcal{L}_{vtm}$. Additionally, we perform multi-semantic adaptive learning mechanism on all visual frames $\boldsymbol{N_v}$ and text tokens $\boldsymbol{N_t}$ through adaptive semantic construction to generate $k$ probabilistic embeddings. Finally, we use $\mathcal{L}_{dst}$ and $\mathcal{L}_{ddsl}$ as constraints to construct multi-instance comparisons within a batch.
• Figure 3: This illustrates MSALM mechanism. The three differently colored shaded areas represent the semantic distributions of two video samples and one text description, reflecting the complex one-to-many relationship between videos and text. The blue and green shades indicate the ambiguous semantic distributions of the videos, while the red shade represents the text distribution. The overlapping areas demonstrate partial matches between the text and multiple videos. The model achieves more flexible and robust semantic matching through multi-point sampling and multi-instance contrastive learning. Light-colored edges emphasize distribution ambiguity.
• Figure 4: Left column: visualization of the USRD drone video-text dataset and UMCRD drone video-text dataset. Right column: visualization word cloud of the (b) USRD dataset and (c) UMCRD dataset.
• Figure 5: Visual analysis of the differences parameter $\lambda$ on USRD drone video-text dataset.