Closer to Ground Truth: Realistic Shape and Appearance Labeled Data Generation for Unsupervised Underwater Image Segmentation

TL;DR

This work tackles underwater fish segmentation under severe visibility constraints and scarce annotations by proposing a two‑stage unsupervised data generation pipeline that blends synthetic fish with real underwater backgrounds. Stage 1 places 2D views of synthetic fish into empty habitats and applies affine, thin plate spline (TPS) warping, and histogram matching to achieve camouflage and realistic deformation; Stage 2 fine‑tunes on data where synthetic fish adopt the appearance of real fish via histogram matching to high‑confidence positives from Stage 1. Evaluated on the established DeepFish dataset and a newly introduced DeepSalmon dataset (30 GB), the method achieves performance close to fully supervised state‑of‑the‑art and can further boost supervised models when used for pseudo‑labeling. The approach reduces labeling effort, demonstrates robustness across diverse underwater scenes, and is generalizable to other non‑rigid objects beyond fish.

Abstract

Solving fish segmentation in underwater videos, a real-world problem of great practical value in marine and aquaculture industry, is a challenging task due to the difficulty of the filming environment, poor visibility and limited existing annotated underwater fish data. In order to overcome these obstacles, we introduce a novel two stage unsupervised segmentation approach that requires no human annotations and combines artificially created and real images. Our method generates challenging synthetic training data, by placing virtual fish in real-world underwater habitats, after performing fish transformations such as Thin Plate Spline shape warping and color Histogram Matching, which realistically integrate synthetic fish into the backgrounds, making the generated images increasingly closer to the real world data with every stage of our approach. While we validate our unsupervised method on the popular DeepFish dataset, obtaining a performance close to a fully-supervised SoTA model, we further show its effectiveness on the specific case of salmon segmentation in underwater videos, for which we introduce DeepSalmon, the largest dataset of its kind in the literature (30 GB). Moreover, on both datasets we prove the capability of our approach to boost the performance of the fully-supervised SoTA model.

Figures (7)

• Figure 1: Examples of synthetic fish placed into images from DeepFish dataset at both stages of our approach. Note how virtual fish naturally fits in the underwater images due to the data transformations performed.
• Figure 2: Thin plate spline image warping transformation examples b) - e), when using random pixel positions displacements for a synthetic fish input image (a). Number of control points is $N=3$.
• Figure 3: Examples of synthetic fish placed into images from DeepFish dataset at both steps of our approach. Note how the generated images become more and more realistic with every stage of our method due to the fish augmentations performed.
• Figure 4: Stage 2 generated data for DeepFish dataset when applying (b) or not (a) Thin-Plate-Spline and color histogram matching virtual fish transformations. Note the unrealistic appearance of the generated images when these augmentations are not used.
• Figure 5: Examples of synthetic fish placed into real underwater fish images in our Realistic Shape and Appearance Data Generation method.