MMOT: The First Challenging Benchmark for Drone-based Multispectral Multi-Object Tracking

TL;DR

This work introduces MMOT, the first large-scale drone-based multispectral MOT benchmark with precise oriented bounding boxes to tackle aerial tracking under challenging small-object, high-density, and occlusion conditions. It couples a multispectral front-end (Spectral 3D-Stem) with orientation-aware state estimation and end-to-end angle-aware tracking to leverage spectral cues and orientation information across existing MOT frameworks. Empirical results show that multispectral inputs significantly improve detection and association, especially for small and densely packed targets, and ablations demonstrate the effectiveness of the Spectral 3D-Stem and orientation components. By releasing the dataset and code, the paper provides a practical foundation to advance orientation-aware, multispectral MOT for real-world UAV deployments.

Abstract

Drone-based multi-object tracking is essential yet highly challenging due to small targets, severe occlusions, and cluttered backgrounds. Existing RGB-based tracking algorithms heavily depend on spatial appearance cues such as color and texture, which often degrade in aerial views, compromising reliability. Multispectral imagery, capturing pixel-level spectral reflectance, provides crucial cues that enhance object discriminability under degraded spatial conditions. However, the lack of dedicated multispectral UAV datasets has hindered progress in this domain. To bridge this gap, we introduce MMOT, the first challenging benchmark for drone-based multispectral multi-object tracking. It features three key characteristics: (i) Large Scale - 125 video sequences with over 488.8K annotations across eight categories; (ii) Comprehensive Challenges - covering diverse conditions such as extreme small targets, high-density scenarios, severe occlusions, and complex motion; and (iii) Precise Oriented Annotations - enabling accurate localization and reduced ambiguity under aerial perspectives. To better extract spectral features and leverage oriented annotations, we further present a multispectral and orientation-aware MOT scheme adapting existing methods, featuring: (i) a lightweight Spectral 3D-Stem integrating spectral features while preserving compatibility with RGB pretraining; (ii) an orientation-aware Kalman filter for precise state estimation; and (iii) an end-to-end orientation-adaptive transformer. Extensive experiments across representative trackers consistently show that multispectral input markedly improves tracking performance over RGB baselines, particularly for small and densely packed objects. We believe our work will advance drone-based multispectral multi-object tracking research. Our MMOT, code, and benchmarks are publicly available at https://github.com/Annzstbl/MMOT.

Figures (13)

• Figure 1: Targets are more distinguishable in multispectral imagery. In the RGB view (left), pedestrians are visually indistinct and overwhelmed by background. In contrast, the MSI view (right) reveals clear spectral separation between targets and the background, highlighting the enhanced discriminability provided by spectral cues.
• Figure 2: Comparison of different annotations in UAV views.
• Figure 3: Example annotations from MMOT showcasing diverse and challenging scenarios. In these scenes, where spatial features are limited due to small object size, clutter or blur, spectral cues provide critical complementary information for reliable discrimination. Zoom in for better visualization.
• Figure 4: Distributions of dataset split, object size, inter-frame IoU of the same object, instances over object categories, and trajectory lengths in MMOT.
• Figure 5: Our proposed Spectral 3D-Stem module employs a Conv3D to extract spectral-spatial features, followed by a depthwise Conv3D to fold the spectral dimension.