Large Language Model Guided Progressive Feature Alignment for Multimodal UAV Object Detection

TL;DR

This work tackles semantic and spatial misalignment in multimodal UAV object detection by introducing LPANet, which uses LLM-derived semantic features from ChatGPT-generated category descriptions encoded by MPNet to guide progressive semantic and spatial alignment across RGB and IR modalities. The framework introduces three modules—Semantic Alignment Module (SAM), Explicit Spatial Alignment Module (ESM), and Implicit Spatial Alignment Module (ISM)—to achieve coarse-to-fine alignment, coupled with a two-stage training regime and a KL-based symmetric consistency loss. Empirical results on DroneVehicle and VEDAI show consistent mAP gains and real-time inference speeds, with ablations confirming the contributions of each module and the LLM-based semantic guidance. The approach demonstrates the practical potential of integrating semantic priors from large language models into multimodal UAV perception to surpass state-of-the-art detectors.

Abstract

Existing multimodal UAV object detection methods often overlook the impact of semantic gaps between modalities, which makes it difficult to achieve accurate semantic and spatial alignments, limiting detection performance. To address this problem, we propose a Large Language Model (LLM) guided Progressive feature Alignment Network called LPANet, which leverages the semantic features extracted from a large language model to guide the progressive semantic and spatial alignment between modalities for multimodal UAV object detection. To employ the powerful semantic representation of LLM, we generate the fine-grained text descriptions of each object category by ChatGPT and then extract the semantic features using the large language model MPNet. Based on the semantic features, we guide the semantic and spatial alignments in a progressive manner as follows. First, we design the Semantic Alignment Module (SAM) to pull the semantic features and multimodal visual features of each object closer, alleviating the semantic differences of objects between modalities. Second, we design the Explicit Spatial alignment Module (ESM) by integrating the semantic relations into the estimation of feature-level offsets, alleviating the coarse spatial misalignment between modalities. Finally, we design the Implicit Spatial alignment Module (ISM), which leverages the cross-modal correlations to aggregate key features from neighboring regions to achieve implicit spatial alignment. Comprehensive experiments on two public multimodal UAV object detection datasets demonstrate that our approach outperforms state-of-the-art multimodal UAV object detectors.

Figures (7)

• Figure 1: Illustration of semantic gap and spatial misalignment issues in multimodal object detection. (a) The semantic gap between different visual modalities causes confusion in the fused region's semantic features, leading to incorrect object category predictions. (b) The semantic gap between visual representations and category labels leads to incorrect object category predictions. (c) Spatial position discrepancies between different visual modalities impact modality fusion.
• Figure 2: The framework diagram of our proposed Large Language Model guided Progressive feature Alignment Network (LPANet) for multi-modal UAV object detection
• Figure 3: The structures of Implicit Spatial alignment Module. The bottom-right corner shows an example of how we calculate the Symmetric Consistency Loss (SR Loss).
• Figure 4: Comparison of speed and accuracy on the DroneVehicle. Speed is compared using FPS.
• Figure 5: Visualization of detection results on the DroneVehicle dataset, with different color boxes representing different categories.