FTDMamba: Frequency-Assisted Temporal Dilation Mamba for Unmanned Aerial Vehicle Video Anomaly Detection

TL;DR

This work tackles UAV video anomaly detection in dynamic backgrounds by modeling multi-source motion with frequency-domain decoupling and multi-scale temporal dynamics. The proposed FTDMamba framework combines a Frequency Decoupled Spatiotemporal Correlation Module (FDSCM) with a Temporal Dilation Mamba Module (TDMM) in a parallel encoder-decoder architecture, enabling robust future-frame prediction under ego-motion and foreground motion. A large moving-UAV dataset, MUVAD, is introduced to benchmark dynamic-background VAD, and FTDMamba achieves state-of-the-art results across three datasets (Drone-Anomaly, UIT-ADrone, and MUVAD) with strong robustness to noise and occlusion. The approach advances practical UAV surveillance by effectively separating global background motion from local object dynamics and by exploiting multi-scale temporal information for reliable anomaly detection in real-world scenarios.

Abstract

Recent advances in video anomaly detection (VAD) mainly focus on ground-based surveillance or unmanned aerial vehicle (UAV) videos with static backgrounds, whereas research on UAV videos with dynamic backgrounds remains limited. Unlike static scenarios, dynamically captured UAV videos exhibit multi-source motion coupling, where the motion of objects and UAV-induced global motion are intricately intertwined. Consequently, existing methods may misclassify normal UAV movements as anomalies or fail to capture true anomalies concealed within dynamic backgrounds. Moreover, many approaches do not adequately address the joint modeling of inter-frame continuity and local spatial correlations across diverse temporal scales. To overcome these limitations, we propose the Frequency-Assisted Temporal Dilation Mamba (FTDMamba) network for UAV VAD, including two core components: (1) a Frequency Decoupled Spatiotemporal Correlation Module, which disentangles coupled motion patterns and models global spatiotemporal dependencies through frequency analysis; and (2) a Temporal Dilation Mamba Module, which leverages Mamba's sequence modeling capability to jointly learn fine-grained temporal dynamics and local spatial structures across multiple temporal receptive fields. Additionally, unlike existing UAV VAD datasets which focus on static backgrounds, we construct a large-scale Moving UAV VAD dataset (MUVAD), comprising 222,736 frames with 240 anomaly events across 12 anomaly types. Extensive experiments demonstrate that FTDMamba achieves state-of-the-art (SOTA) performance on two public static benchmarks and the new MUVAD dataset. The code and MUVAD dataset will be available at: https://github.com/uavano/FTDMamba.

Figures (10)

• Figure 1: Illustration of the multi-source motion coupling issue. (a) and (b) show the frame differences between UAV videos with dynamic backgrounds and ground-based surveillance videos with static backgrounds.
• Figure 2: Comparison of spectral distributions for global and local motion patterns. (a) Pure global motion: average frequency spectrum computed from 30 frames containing only UAV-induced motion. (b) Pure local motion: average frequency spectrum computed from 30 frames with static backgrounds and exclusively moving objects.
• Figure 3: Overview of the proposed FTDMamba. (a) The overall pipeline adopts an encoder-decoder architecture that contains two core parallel modules: the Frequency Decoupled Spatiotemporal Correlation Module (FDSCM) and the Temporal Dilation Mamba Module (TDMM). (b) The FDSCM first employs a 1D FFT on the temporal dimension for initial multi-source motion decoupling, and subsequently utilizes a 2D FFT on the spatiotemporal domain to model global spatiotemporal dependencies. (c) The TDMM leverages the core STMamba module (d) which constructs distinct pixel-wise temporal-first and patch-wise spatial-first scanning sequences, jointly modeling temporal continuity and local spatial correlations from multiple temporal scales.
• Figure 4: Illustration of the scanning strategy in STMamba. (a) flattens spatial dimensions in row-major and column-major orders, then concatenates along the temporal axis. (b) divides frame features into patches and extracts temporal sequences at each patch location, then concatenates them in row-major and column-major orders. Here, we only show forward scanning.
• Figure 5: Multi-scale temporal modeling strategy. This strategy processes features at different temporal scales by reorganizing consecutive frame features into subsequences with varying temporal resolutions.