How to Track and Segment Fish without Human Annotations: A Self-Supervised Deep Learning Approach

TL;DR

This work tackles the problem of tracking and segmenting fish in unconstrained underwater videos without manual annotations. It introduces a three-stage self-supervised framework that first generates pseudo-labels via background subtraction and optical flow, then refines these labels with a historical moving-average mechanism and CRF sharpening, and finally trains a SOLOv2-based segmentation model alongside online tracking with SORT. Experiments on Seagrass Ditria, DeepFish, and YouTube-VOS demonstrate competitive unsupervised performance closely approaching fully supervised baselines while maintaining robustness across diverse imaging conditions. The approach reduces labeling costs and offers a scalable tool for ecological monitoring and fish behavior analysis in natural habitats.

Abstract

Tracking fish movements and sizes of fish is crucial to understanding their ecology and behaviour. Knowing where fish migrate, how they interact with their environment, and how their size affects their behaviour can help ecologists develop more effective conservation and management strategies to protect fish populations and their habitats. Deep learning is a promising tool to analyze fish ecology from underwater videos. However, training deep neural networks (DNNs) for fish tracking and segmentation requires high-quality labels, which are expensive to obtain. We propose an alternative unsupervised approach that relies on spatial and temporal variations in video data to generate noisy pseudo-ground-truth labels. We train a multitask DNN using these pseudo-labels. Our framework consists of three stages: (1) an optical flow model generates the pseudo labels using spatial and temporal consistency between frames, (2) a self-supervised model refines the pseudo-labels incrementally, and (3) a segmentation network uses the refined labels for training. Consequently, we perform extensive experiments to validate our method on three public underwater video datasets and demonstrate its effectiveness for video annotation and segmentation. We also evaluate its robustness to different imaging conditions and discuss its limitations.

Figures (11)

• Figure 1: Combining background subtraction and optical flow demonstrate how both levels work in concert to preserve object boundaries and temporal coherence throughout the video. Please refer to \ref{['secfram']} for details.
• Figure 2: Our proposed framework consists of three main components: generate pseudo-labels, unsupervised pseudo-labels refinement, and segmentation network. The proposed segmentation model trains with the generated pseudo-labels, which are refined with self-supervised training. Please refer to \ref{['secfram']} for details.
• Figure 3: Sample image from each of the four utilised datasets. From left: Seagrass Ditria2021a, DeepFish Saleh2020a, YouTube-VOS Xu2018b, and Mediterranean Fish Species MediterraneanFish
• Figure 4: Sample optical flow results for Seagrass Ditria2021a. From left, the original image, optical flow without background subtraction, optical flow with background subtraction, mask overlay.
• Figure 5: Sample optical flow results for DeepFish Saleh2020a. From left, the original image, optical flow without background subtraction, optical flow with background subtraction, mask overlay.