When Trackers Date Fish: A Benchmark and Framework for Underwater Multiple Fish Tracking

TL;DR

The paper addresses the problem of underwater multiple fish tracking by introducing MFT25, a large-scale, richly annotated underwater MOT dataset, and SU-T, a specialized baseline that combines a YOLOX-based detector, optional Re-ID, and a UKF-driven motion model with FishIoU-based association. Key contributions include the FishIoU metric designed for elongated fish morphology, a three-stage cascade association, and extensive experiments showing state-of-the-art performance on MFT25 (e.g., 34.1 HOTA, 44.6 IDF1) while revealing fundamental differences from terrestrial tracking. The work demonstrates that underwater tracking requires domain-specific adaptations in motion modeling, appearance cues, and association strategies, with practical implications for marine ecology and aquaculture. Overall, MFT25 and SU-T provide a robust foundation for advancing underwater MFT research and benchmarking future methods in challenging aquatic environments.

Abstract

Multiple object tracking (MOT) technology has made significant progress in terrestrial applications, but underwater tracking scenarios remain underexplored despite their importance to marine ecology and aquaculture. In this paper, we present Multiple Fish Tracking Dataset 2025 (MFT25), a comprehensive dataset specifically designed for underwater multiple fish tracking, featuring 15 diverse video sequences with 408,578 meticulously annotated bounding boxes across 48,066 frames. Our dataset captures various underwater environments, fish species, and challenging conditions including occlusions, similar appearances, and erratic motion patterns. Additionally, we introduce Scale-aware and Unscented Tracker (SU-T), a specialized tracking framework featuring an Unscented Kalman Filter (UKF) optimized for non-linear swimming patterns of fish and a novel Fish-Intersection-over-Union (FishIoU) matching that accounts for the unique morphological characteristics of aquatic species. Extensive experiments demonstrate that our SU-T baseline achieves state-of-the-art performance on MFT25, with 34.1 HOTA and 44.6 IDF1, while revealing fundamental differences between fish tracking and terrestrial object tracking scenarios. The dataset and codes are released at https://vranlee.github.io/SU-T/.

Figures (11)

• Figure 1: Evaluation of various MFT and MOT methods on MFT25 benchmark. Detailed results are provided in Table \ref{['tab:tracking-comparison']}.
• Figure 2: Distribution of target movement directions across datasets. Directional instability is notably more pronounced in the fish dataset compared to other target categories.
• Figure 3: The challenges of multiple fish tracking arise from factors such as fish physiological features and the complexity of underwater scenarios.
• Figure 4: The framework of SU-T. The pipeline consists of three main components: a detector, an association module, and an optional Re-ID module. The detector generates bounding boxes and confidence scores for each frame. The optional Re-ID module extracts feature embeddings to enhance tracking accuracy. The association module uses the FishIoU to calculate matching costs between detected boxes and predicted boxes from the UKF.
• Figure 5: Unscented Kalman Filter (UKF) motion model used in SU-T for predicting fish movement in complex underwater environments.