Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations

Quang Trung Truong, Wong Yuk Kwan, Duc Thanh Nguyen, Binh-Son Hua, Sai-Kit Yeung

TL;DR

AUTV presents a unified diffusion-based framework to synthesize underwater videos with pixel-wise annotations from text prompts. By coupling text-to-mask (T2M), mask-to-image (M2I), and image-to-video (I2V) generation with frame-wise temporal conditioning and SAM2-driven mask propagation, AUTV achieves temporally coherent video content aligned to annotations. The framework yields two datasets: UTV, a real-world 2,000 video-text corpus with rich annotations, and SUTV, a synthetic 10,000-video dataset with per-frame segmentation masks. Experiments show that synthetic data from SUTV improves video inpainting and video object segmentation, while UTV enables marine-domain fine-tuning of the generation models, resulting in higher fidelity text-video alignment and annotation quality. Overall, AUTV provides scalable, annotation-rich underwater data for advancing marine computer vision and ecological analysis.

Abstract

Underwater video analysis, hampered by the dynamic marine environment and camera motion, remains a challenging task in computer vision. Existing training-free video generation techniques, learning motion dynamics on the frame-by-frame basis, often produce poor results with noticeable motion interruptions and misaligments. To address these issues, we propose AUTV, a framework for synthesizing marine video data with pixel-wise annotations. We demonstrate the effectiveness of this framework by constructing two video datasets, namely UTV, a real-world dataset comprising 2,000 video-text pairs, and SUTV, a synthetic video dataset including 10,000 videos with segmentation masks for marine objects. UTV provides diverse underwater videos with comprehensive annotations including appearance, texture, camera intrinsics, lighting, and animal behavior. SUTV can be used to improve underwater downstream tasks, which are demonstrated in video inpainting and video object segmentation.

AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations

TL;DR

AUTV presents a unified diffusion-based framework to synthesize underwater videos with pixel-wise annotations from text prompts. By coupling text-to-mask (T2M), mask-to-image (M2I), and image-to-video (I2V) generation with frame-wise temporal conditioning and SAM2-driven mask propagation, AUTV achieves temporally coherent video content aligned to annotations. The framework yields two datasets: UTV, a real-world 2,000 video-text corpus with rich annotations, and SUTV, a synthetic 10,000-video dataset with per-frame segmentation masks. Experiments show that synthetic data from SUTV improves video inpainting and video object segmentation, while UTV enables marine-domain fine-tuning of the generation models, resulting in higher fidelity text-video alignment and annotation quality. Overall, AUTV provides scalable, annotation-rich underwater data for advancing marine computer vision and ecological analysis.

Abstract

Underwater video analysis, hampered by the dynamic marine environment and camera motion, remains a challenging task in computer vision. Existing training-free video generation techniques, learning motion dynamics on the frame-by-frame basis, often produce poor results with noticeable motion interruptions and misaligments. To address these issues, we propose AUTV, a framework for synthesizing marine video data with pixel-wise annotations. We demonstrate the effectiveness of this framework by constructing two video datasets, namely UTV, a real-world dataset comprising 2,000 video-text pairs, and SUTV, a synthetic video dataset including 10,000 videos with segmentation masks for marine objects. UTV provides diverse underwater videos with comprehensive annotations including appearance, texture, camera intrinsics, lighting, and animal behavior. SUTV can be used to improve underwater downstream tasks, which are demonstrated in video inpainting and video object segmentation.

Paper Structure

This paper contains 21 sections, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Text-to-Video (T2V)
  4. Training-free T2V
  5. AUTV framework
  6. Overview
  7. T2M generation
  8. M2I generation
  9. I2V generation
  10. Video mask generation
  11. Datasets
  12. UTV - a real-world video-text dataset with fine-grained annotations
  13. SUTV - a synthetic video dataset with pixel-wise annotations
  14. Experiments
  15. Experimental setup
  16. ...and 6 more sections

Figures (10)

  • Figure 1: Given masks of the first video frame and a text, or a text, our method synthesizes a high-fidelity video
  • Figure 2: Generated video object masks (first row) and corresponding video frames (second row) by applying SegGen ye2023seggen in a frame-wise basis. As show, there is a huge discrepancy between the generated masks and video frames, due to a lack of integration of motion during the synthesis process.
  • Figure 3: Overview of our AUTV framework. (a) Video generation pipeline. (b) Image2Video module built on a video diffusion model (VDM), i.e., ModelScopeT2V wang2023modelscope. Please zoom-in for the best view.
  • Figure 4: Our video captioning annotation. (a) Our captioning pipeline. (b) An example of annotated attributes. Please see the supplementary material for more examples.
  • Figure 5: Word cloud of the text in real-world dataset (UTV).
  • ...and 5 more figures