Deep Learning for Accurate Vision-based Catch Composition in Tropical Tuna Purse Seiners

TL;DR

This study tackles the challenge of estimating species-specific catch composition from electronic monitoring (EM) footage in tropical tuna purse seiners, where expert agreement on BET versus YFT is low. By building a four-stage pipeline (data collection, labeling, model training/validation, and ground-truth testing) and leveraging YOLOv9+SAM2 for segmentation combined with hierarchical classification, the authors achieve robust performance on real-world operations. Ground-truth onboard identifications address ground-truth bias and enable meaningful evaluation with artificial fishing operations (AFOs). The results show that the best-performing configuration segments and classifies a large majority of visible fish with low mean absolute error, and that camera type (global vs rolling shutter) strongly affects accuracy, underscoring practical considerations for deployment and scaling to other fisheries.

Abstract

Purse seiners play a crucial role in tuna fishing, as approximately 69% of the world's tropical tuna is caught using this gear. All tuna Regional Fisheries Management Organizations have established minimum standards to use electronic monitoring (EM) in fisheries in addition to traditional observers. The EM systems produce a massive amount of video data that human analysts must process. Integrating artificial intelligence (AI) into their workflow can decrease that workload and improve the accuracy of the reports. However, species identification still poses significant challenges for AI, as achieving balanced performance across all species requires appropriate training data. Here, we quantify the difficulty experts face to distinguish bigeye tuna (BET, Thunnus Obesus) from yellowfin tuna (YFT, Thunnus Albacares) using images captured by EM systems. We found inter-expert agreements of 42.9% $\pm$ 35.6% for BET and 57.1% $\pm$ 35.6% for YFT. We then present a multi-stage pipeline to estimate the species composition of the catches using a reliable ground-truth dataset based on identifications made by observers on board. Three segmentation approaches are compared: Mask R-CNN, a combination of DINOv2 with SAM2, and a integration of YOLOv9 with SAM2. We found that the latest performs the best, with a validation mean average precision of 0.66 $\pm$ 0.03 and a recall of 0.88 $\pm$ 0.03. Segmented individuals are tracked using ByteTrack. For classification, we evaluate a standard multiclass classification model and a hierarchical approach, finding a superior generalization by the hierarchical. All our models were cross-validated during training and tested on fishing operations with fully known catch composition. Combining YOLOv9-SAM2 with the hierarchical classification produced the best estimations, with 84.8% of the individuals being segmented and classified with a mean average error of 4.5%.

Figures (5)

• Figure 1: Workflow that outlines the pipeline for fish image analysis. It begins with the collection and manual annotation of images, followed by model training using cross-validation to select the best-performing architecture. The selected model is then used to automatically segment and classify fish in images. Finally, the pipeline is tested on real-world data. This process ensures robust and accurate fish identification.
• Figure 2: Manually annotated image. This image contains a masks for each fish on the conveyor belt that is visible from the point of view of the camera. This frame belongs to a monospecific SKJ sample. It was used during the training of the models.
• Figure 3: Different statistics for the analysis based on expert identification.
• Figure 4: Comparison of all the segmentation approaches. It can be noted that the right part of the camera has a small water droplet, making the fish in that area blurry and more difficult to segment.
• Figure 5: Confusion matrices for the four classification models. These matrices are built using the validation data.