Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UVLM: Benchmarking Video Language Model for Underwater World Understanding

Xizhe Xue, Yang Zhou, Dawei Yan, Lijie Tao, Junjie Li, Ying Li, Haokui Zhang, Rong Xiao

TL;DR

This work introduces UVLM, the first underwater video-language benchmark designed to push multimedia models beyond terrestrial domains. By combining a dual-path data collection with a human–AI annotation pipeline (including GPT-4o-assisted QA) and 20 scientifically motivated subtasks across 419 marine taxa, UVLM enables fine-grained temporal and ecological reasoning in underwater videos. The dataset comprises 2,109 videos (≈0.86M frames) and roughly 40k video–text pairs, with a comprehensive evaluation protocol featuring two objective metrics and five LLM-based judgments. Empirical results show that fine-tuning VidLMs on UVLM substantially improves underwater understanding and can also yield gains on in-air benchmarks, supporting advances in marine science and autonomous underwater perception; the dataset and prompts will be released publicly for broad community use.

Abstract

Recently, the remarkable success of large language models (LLMs) has achieved a profound impact on the field of artificial intelligence. Numerous advanced works based on LLMs have been proposed and applied in various scenarios. Among them, video language models (VidLMs) are particularly widely used. However, existing works primarily focus on terrestrial scenarios, overlooking the highly demanding application needs of underwater observation. To overcome this gap, we introduce UVLM, an under water observation benchmark which is build through a collaborative approach combining human expertise and AI models. To ensure data quality, we have conducted in-depth considerations from multiple perspectives. First, to address the unique challenges of underwater environments, we selected videos that represent typical underwater challenges including light variations, water turbidity, and diverse viewing angles to construct the dataset. Second, to ensure data diversity, the dataset covers a wide range of frame rates, resolutions, 419 classes of marine animals, and various static plants and terrains. Next, for task diversity, we adopted a structured design where observation targets are categorized into two major classes: biological and environmental. Each category includes content observation and change/action observation, totaling 20 distinct task types. Finally, we designed several challenging evaluation metrics to enable quantitative comparison and analysis of different methods. Experiments on two representative VidLMs demonstrate that fine-tuning VidLMs on UVLM significantly improves underwater world understanding while also showing potential for slight improvements on existing in-air VidLM benchmarks, such as VideoMME and Perception text. The dataset and prompt engineering will be released publicly.

UVLM: Benchmarking Video Language Model for Underwater World Understanding

TL;DR

This work introduces UVLM, the first underwater video-language benchmark designed to push multimedia models beyond terrestrial domains. By combining a dual-path data collection with a human–AI annotation pipeline (including GPT-4o-assisted QA) and 20 scientifically motivated subtasks across 419 marine taxa, UVLM enables fine-grained temporal and ecological reasoning in underwater videos. The dataset comprises 2,109 videos (≈0.86M frames) and roughly 40k video–text pairs, with a comprehensive evaluation protocol featuring two objective metrics and five LLM-based judgments. Empirical results show that fine-tuning VidLMs on UVLM substantially improves underwater understanding and can also yield gains on in-air benchmarks, supporting advances in marine science and autonomous underwater perception; the dataset and prompts will be released publicly for broad community use.

Abstract

Recently, the remarkable success of large language models (LLMs) has achieved a profound impact on the field of artificial intelligence. Numerous advanced works based on LLMs have been proposed and applied in various scenarios. Among them, video language models (VidLMs) are particularly widely used. However, existing works primarily focus on terrestrial scenarios, overlooking the highly demanding application needs of underwater observation. To overcome this gap, we introduce UVLM, an under water observation benchmark which is build through a collaborative approach combining human expertise and AI models. To ensure data quality, we have conducted in-depth considerations from multiple perspectives. First, to address the unique challenges of underwater environments, we selected videos that represent typical underwater challenges including light variations, water turbidity, and diverse viewing angles to construct the dataset. Second, to ensure data diversity, the dataset covers a wide range of frame rates, resolutions, 419 classes of marine animals, and various static plants and terrains. Next, for task diversity, we adopted a structured design where observation targets are categorized into two major classes: biological and environmental. Each category includes content observation and change/action observation, totaling 20 distinct task types. Finally, we designed several challenging evaluation metrics to enable quantitative comparison and analysis of different methods. Experiments on two representative VidLMs demonstrate that fine-tuning VidLMs on UVLM significantly improves underwater world understanding while also showing potential for slight improvements on existing in-air VidLM benchmarks, such as VideoMME and Perception text. The dataset and prompt engineering will be released publicly.

Paper Structure

This paper contains 24 sections, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Language Understanding Benchmark
  4. Underwater Observation Datasets
  5. UVLM
  6. Overview
  7. Data Collection and Annotation
  8. UVLM Statistics
  9. Evaluation Metrics
  10. Experiments
  11. Dataset Partition
  12. Experimental Results on UVLM
  13. Conclusion
  14. Appendix Contents & Benchmark Link
  15. A. Sample Generate Prompt Details
  16. ...and 9 more sections

Figures (10)

  • Figure 1: Challenges for VidLMs for understanding underwater videos. Our UVLM is proposed to overcome this gap and it enables the 7B VidLM to achieve performance comparable to closed-source models like GPT-4o and Gemini.
  • Figure 1: Dialogue generation prompt
  • Figure 2: An overview of data preparation and generation pipeline for UVLM.
  • Figure 2: Task structure and the examples of UVLM. Left side: Task structure; Right side: several cases
  • Figure 3: Statistics on UVLM. (a) Scene distribution; (b) Observation target distribution; (c) Several samples in UVLM; (d) Fine-grained taxonomic classification information (partial categories), from left to right: Kingdom, Phylum, Class, Order.
  • ...and 5 more figures