Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge

TL;DR

The paper tackles robust UAV detection, classification, and 3D tracking under challenging conditions by introducing a multi-modal framework that fuses camera, LiDAR, and radar data. It splits the system into a UAV type classification branch with sequence fusion, ROI cropping, and keyframe-based soft voting, and a LiDAR/radar-driven pose estimation branch with dynamic point analysis, multi-object tracking, and trajectory completion via an autoregressive model and B-spline smoothing. Extensive experiments on the MMUAD/MMAUD dataset demonstrate state-of-the-art performance in both UAV type classification and 3D pose tracking, achieving first place in the UG2+ CVPR 2024 challenge. The work highlights the practicality of unsupervised dynamic-point analysis and sequence-based fusion for robust multi-modal anti-UAV perception, with potential impact on real-world surveillance and security applications.

Abstract

This technical report presents the 1st winning model for UG2+, a task in CVPR 2024 UAV Tracking and Pose-Estimation Challenge. This challenge faces difficulties in drone detection, UAV-type classification and 2D/3D trajectory estimation in extreme weather conditions with multi-modal sensor information, including stereo vision, various Lidars, Radars, and audio arrays. Leveraging this information, we propose a multi-modal UAV detection, classification, and 3D tracking method for accurate UAV classification and tracking. A novel classification pipeline which incorporates sequence fusion, region of interest (ROI) cropping, and keyframe selection is proposed. Our system integrates cutting-edge classification techniques and sophisticated post-processing steps to boost accuracy and robustness. The designed pose estimation pipeline incorporates three modules: dynamic points analysis, a multi-object tracker, and trajectory completion techniques. Extensive experiments have validated the effectiveness and precision of our approach. In addition, we also propose a novel dataset pre-processing method and conduct a comprehensive ablation study for our design. We finally achieved the best performance in the classification and tracking of the MMUAD dataset. The code and configuration of our method are available at https://github.com/dtc111111/Multi-Modal-UAV.

Figures (7)

• Figure 1: Overview of multi-modal UAV classification and tracking framework.
• Figure 2: The UAV pose estimation pipeline. Initially, we accumulate and cluster data over 20 frames, then proceed with an analysis of the dynamic point cloud to segregate background points from dynamic points. Subsequently, we track these dynamic points and perform center regression on dynamic clusters to derive an initial estimation of the UAV trajectory. The trajectory is further refined and extrapolated using a Kalman filter, culminating in a comprehensive and precise UAV trajectory.
• Figure 3: The pipeline of dynamic point analysis module. We first accumulate 20 frames of point clouds as a temporal window. Then we use an unsupervised clustering method to cluster the point clouds and extract the feature of each group. The LSTM module with MLP head can finally decompose the dynamic points.
• Figure 4: We visualize the distribution of images in the test datasets.
• Figure 5: Some corner case in images.