UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understanding

TL;DR

UWBench addresses the gap in underwater vision-language understanding by introducing a large-scale, expert-verified benchmark consisting of 15,003 images with ecologically informed captions, 15,281 object referring expressions, and 124,983 QA pairs across 158 categories. The authors present a five-stage construction pipeline combining multi-source data collection, attribute extraction, prompt engineering, GPT-5 assisted annotation, and rigorous marine-biologist verification to produce high-quality multimodal data. They define three interconnected evaluation tasks—detailed image captioning, visual grounding, and visual question answering—and benchmark a wide range of state-of-the-art VLMs, revealing substantial domain gaps when compared to terrestrial benchmarks. The results underscore the need for domain-specific adaptations and expert knowledge integration to enable reliable underwater scene understanding, with direct implications for marine science, ecological monitoring, and autonomous underwater exploration.

Abstract

Large vision-language models (VLMs) have achieved remarkable success in natural scene understanding, yet their application to underwater environments remains largely unexplored. Underwater imagery presents unique challenges including severe light attenuation, color distortion, and suspended particle scattering, while requiring specialized knowledge of marine ecosystems and organism taxonomy. To bridge this gap, we introduce UWBench, a comprehensive benchmark specifically designed for underwater vision-language understanding. UWBench comprises 15,003 high-resolution underwater images captured across diverse aquatic environments, encompassing oceans, coral reefs, and deep-sea habitats. Each image is enriched with human-verified annotations including 15,281 object referring expressions that precisely describe marine organisms and underwater structures, and 124,983 question-answer pairs covering diverse reasoning capabilities from object recognition to ecological relationship understanding. The dataset captures rich variations in visibility, lighting conditions, and water turbidity, providing a realistic testbed for model evaluation. Based on UWBench, we establish three comprehensive benchmarks: detailed image captioning for generating ecologically informed scene descriptions, visual grounding for precise localization of marine organisms, and visual question answering for multimodal reasoning about underwater environments. Extensive experiments on state-of-the-art VLMs demonstrate that underwater understanding remains challenging, with substantial room for improvement. Our benchmark provides essential resources for advancing vision-language research in underwater contexts and supporting applications in marine science, ecological monitoring, and autonomous underwater exploration. Our code and benchmark will be available.

Figures (10)

• Figure 1: Challenges for VLMs for understanding underwater scenes.
• Figure 2: Examples of an image and corresponding annotations in UWBench dataset. Our annotations include object referring, visual question answering, and detailed captions.
• Figure 3: Overview of UWBench Construction. The pipeline starts with multi-source underwater image acquisition—including web search, public dataset selection, and real-world in-situ shooting—followed by refined attribute extraction that structurally organizes image and object-level features such as resolution, habitat type, water quality, illumination, taxonomy, spatial, and morphological information. Comprehensive prompt engineering then guides GPT-5 to automatically generate detailed captions, distinctive referring expressions, and diverse visual QA pairs. Finally, annotation quality is verified and enhanced through a three-stage process—primary review, expert assessment, and cross-validation—resulting in a scientifically rigorous, ecologically informative, and broadly representative underwater vision-language dataset.
• Figure 4: Statistics of the UWBench caption dataset. (a) Overall Vocabulary Word Cloud. (b) Underwater Domain-Specific Terms. (c) Top 20 Words Co-occurrence Heatmap. (d) Top 30 Words Radial View. (e) Word vs Character Count Scatter. (f) Word Count Distribution. (g) Word Association Network.
• Figure 5: Statistics of object referring of UWBench. (a) Word Cloud of Categories. (b) Object Size by Category (Top 15). (c) Density Heatmap for BBox Dimensions. (d) BBox Area Distribution. (e) Referring Sentence Length Distribution. (f) Top 10 Categories Pie Chart (Relative Proportion).