Exploring the Underwater World Segmentation without Extra Training

TL;DR

This work tackles the lack of fine-grained underwater segmentation resources and practical domain adaptation by introducing AquaOV255, a large-scale underwater dataset with 255 categories and 20,723 images, and UOVSBench for unified open-vocabulary evaluation. It then presents Earth2Ocean, a training-free framework that transfers terrestrial vision-language models to underwater domains via a geometry-guided visual mask generator and a multimodal, reasoning-enhanced category-visual alignment module. The approach yields consistent improvements over state-of-the-art training-free methods across multiple backbones and datasets, while maintaining efficient inference. The resources and methodology offer a practical path for marine biodiversity monitoring and ecological analysis without underwater model retraining.

Abstract

Accurate segmentation of marine organisms is vital for biodiversity monitoring and ecological assessment, yet existing datasets and models remain largely limited to terrestrial scenes. To bridge this gap, we introduce \textbf{AquaOV255}, the first large-scale and fine-grained underwater segmentation dataset containing 255 categories and over 20K images, covering diverse categories for open-vocabulary (OV) evaluation. Furthermore, we establish the first underwater OV segmentation benchmark, \textbf{UOVSBench}, by integrating AquaOV255 with five additional underwater datasets to enable comprehensive evaluation. Alongside, we present \textbf{Earth2Ocean}, a training-free OV segmentation framework that transfers terrestrial vision--language models (VLMs) to underwater domains without any additional underwater training. Earth2Ocean consists of two core components: a Geometric-guided Visual Mask Generator (\textbf{GMG}) that refines visual features via self-similarity geometric priors for local structure perception, and a Category-visual Semantic Alignment (\textbf{CSA}) module that enhances text embeddings through multimodal large language model reasoning and scene-aware template construction. Extensive experiments on the UOVSBench benchmark demonstrate that Earth2Ocean achieves significant performance improvement on average while maintaining efficient inference.

Figures (14)

• Figure 1: Our contributions: (a) proposing a fine-grained and diverse underwater dataset and benchmark, and (b) designing a practical training-free framework that enables zero-training adaptation for underwater scene transfer.
• Figure 2: (a.1) Compares image and category counts across datasets (AquaOV255 has about 61% category growth); (a.2) Visualizes category distribution via word cloud; (b.1) Evaluates method effectiveness (aAcc, mIoU, mAcc); (b.2) Trades off mIoU and FPS.
• Figure 3: Foreground Part: On the left, we show the AquaOV255 dataset along with its fine-grained splits; on the right, we present the UOVSBench Benchmark, including the datasets and image category counts. Background Part: Examples Visualization from Our AquaOV255 Dataset. For further numerical analysis and properties, please refer to the Appendix.
• Figure 4: This diagram outlines the Earth2Ocean framework, which integrates (a) Geometric-guided Visual Mask Generator (GMG) to produce locally consistent and geometry-aware masks, (b) Category-visual Semantic Alignment (CSA) leveraging MLLM reasoning to improve underwater category alignment, and (c) mask classification for final pixel-level prediction. Together, these components enable training-free, open-vocabulary segmentation adapted to underwater scenes.
• Figure 5: MLLMs demonstrate superior capability over VLMs in capturing semantic and visual cues for underwater object classification accuracy (Y-axis).