A Multimodal Transformer Approach for UAV Detection and Aerial Object Recognition Using Radar, Audio, and Video Data

TL;DR

The paper addresses UAV detection and aerial object recognition under complex environments by proposing a multimodal Transformer that fuses radar, RGB video, IR video, and audio. It introduces an end-to-end pipeline with cyclical replication, feature extraction, and normalization to produce a unified input for a Transformer with early fusion, achieving high macro-averaged metrics and real-time inference. The work demonstrates state-of-the-art performance on independent test data and provides a detailed account of data handling, architecture, and evaluation, highlighting practical implications for surveillance and security. The results suggest robust, cross-modal representations that enhance UAV detection reliability while maintaining computational efficiency suitable for near-real-time deployment.

Abstract

Unmanned aerial vehicle (UAV) detection and aerial object recognition are critical for modern surveillance and security, prompting a need for robust systems that overcome limitations of single-modality approaches. This research addresses these challenges by designing and rigorously evaluating a novel multimodal Transformer model that integrates diverse data streams: radar, visual band video (RGB), infrared (IR) video, and audio. The architecture effectively fuses distinct features from each modality, leveraging the Transformer's self-attention mechanisms to learn comprehensive, complementary, and highly discriminative representations for classification. The model demonstrated exceptional performance on an independent test set, achieving macro-averaged metrics of 0.9812 accuracy, 0.9873 recall, 0.9787 precision, 0.9826 F1-score, and 0.9954 specificity. Notably, it exhibited particularly high precision and recall in distinguishing drones from other aerial objects. Furthermore, computational analysis confirmed its efficiency, with 1.09 GFLOPs, 1.22 million parameters, and an inference speed of 41.11 FPS, highlighting its suitability for real-time applications. This study presents a significant advancement in aerial object classification, validating the efficacy of multimodal data fusion via a Transformer architecture for achieving state-of-the-art performance, thereby offering a highly accurate and resilient solution for UAV detection and monitoring in complex airspace.

Figures (7)

• Figure 1: This diagram illustrates the sequential and parallel steps involved in preparing data for a multimodal Transformer model. It includes data loading, cyclical replication, feature extraction, standardization, normalization, and final tensor storage.
• Figure 2: Audio and radar spectrograms, and raw frames from RGB and IR video modalities.
• Figure 3: Data distribution comparison between Raw Original and Raw Replicated (Augmented) for various modalities.
• Figure 4: Data distribution comparison between Raw Original and Processed Features for various modalities.
• Figure 5: This diagram illustrates the forward pass of the multimodal Transformer, detailing how audio, IR video, RGB video, and radar inputs are processed through individual projections, positional encoding, a shared Transformer encoder, and a final classification head to predict the target class.