MambaNUT: Nighttime UAV Tracking via Mamba-based Adaptive Curriculum Learning

TL;DR

Nighttime UAV tracking suffers from low illumination and day–night domain gaps, and existing ViT-based trackers are often too resource-intensive for onboard deployment. This work introduces MambaNUT, a pure Mamba-based end-to-end tracker with a linear-complexity state-space backbone and a one-stream Vision Mamba architecture, augmented by Adaptive Curriculum Learning (ACL) to bridge day/night data. ACL comprises a sampling scheduler that gradually balances day and night samples and a data-weighted loss (ADW) that emphasizes hard/night cases via $U_i$ and dataset sizes, guided by $\omega_i=\log(N_{max}/N_j)+0.5$. Across NAT2024-1, NAT2021, and UAVDark135, MambaNUT delivers state-of-the-art nighttime tracking performance at high efficiency (approximately 75 FPS, 1.1 GMac, 4.1M parameters), validating end-to-end, low-latency tracking on resource-constrained platforms.

Abstract

Harnessing low-light enhancement and domain adaptation, nighttime UAV tracking has made substantial strides. However, over-reliance on image enhancement, limited high-quality nighttime data, and a lack of integration between daytime and nighttime trackers hinder the development of an end-to-end trainable framework. Additionally, current ViT-based trackers demand heavy computational resources due to their reliance on the self-attention mechanism. In this paper, we propose a novel pure Mamba-based tracking framework (MambaNUT) that employs a state space model with linear complexity as its backbone, incorporating a single-stream architecture that integrates feature learning and template-search coupling within Vision Mamba. We introduce an adaptive curriculum learning (ACL) approach that dynamically adjusts sampling strategies and loss weights, thereby improving the model's ability of generalization. Our ACL is composed of two levels of curriculum schedulers: (1) sampling scheduler that transforms the data distribution from imbalanced to balanced, as well as from easier (daytime) to harder (nighttime) samples; (2) loss scheduler that dynamically assigns weights based on the size of the training set and IoU of individual instances. Exhaustive experiments on multiple nighttime UAV tracking benchmarks demonstrate that the proposed MambaNUT achieves state-of-the-art performance while requiring lower computational costs. The code will be available at https://github.com/wuyou3474/MambaNUT.

Figures (7)

• Figure 1: Compared to SOTA trackers on NAT2024-1 fu2024prompt, our MambaNUT sets a new record with 83.3% precision and a speed of 75 FPS, while requiring the lowest computational cost. Note that bubble size reflects the number of parameters, with larger bubbles indicating more parameters.
• Figure 2: Training data distribution varies sharply between daytime and nighttime datasets.
• Figure 3: Overview of the proposed MambaNUT framework. It includes a Vision Mamba backbone and a tracking head, integrating an adaptive curriculum learning (ACL) approach with two schedulers: (1) a sampling scheduler that balances the data distribution from easier (daytime) to harder (nighttime) samples, and (2) a loss scheduler that assigns weights based on training data size and IoU of individual instances.
• Figure 4: Qualitative evaluation on two video sequences from NAT2024-1: L50001 and L50011.
• Figure 5: Illumination-oriented evaluation comparison with the 8 SOTA trackers, evaluated on NAT2024-1fu2024prompt.