Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ORCA: Object Recognition and Comprehension for Archiving Marine Species

Yuk-Kwan Wong, Haixin Liang, Zeyu Ma, Yiwei Chen, Ziqiang Zheng, Rinaldi Gotama, Pascal Sebastian, Lauren D. Sparks, Sai-Kit Yeung

TL;DR

ORCA addresses the lack of standardized, domain-specific benchmarks for marine vision by assembling a large-scale multimodal dataset with 478 species, 14,647 images, 42,217 bounding boxes, and 22,321 expert-verified instance captions. It enables three core tasks—object detection (closed-set and open-vocabulary), instance captioning, and visual grounding—and introduces taxonomy-aware evaluation settings to address morphological overlap. A broad benchmark of 18 state-of-the-art models reveals that domain-specific supervision and dense, instance-level captions substantially improve localization and description, while generic vision-language models struggle to generate marine-specific, region-level outputs. ORCA thus provides a concrete, ecologically relevant platform to advance marine visual understanding and archiving, with practical implications for biodiversity monitoring and conservation through more accurate identification and documentation of marine species.

Abstract

Marine visual understanding is essential for monitoring and protecting marine ecosystems, enabling automatic and scalable biological surveys. However, progress is hindered by limited training data and the lack of a systematic task formulation that aligns domain-specific marine challenges with well-defined computer vision tasks, thereby limiting effective model application. To address this gap, we present ORCA, a multi-modal benchmark for marine research comprising 14,647 images from 478 species, with 42,217 bounding box annotations and 22,321 expert-verified instance captions. The dataset provides fine-grained visual and textual annotations that capture morphology-oriented attributes across diverse marine species. To catalyze methodological advances, we evaluate 18 state-of-the-art models on three tasks: object detection (closed-set and open-vocabulary), instance captioning, and visual grounding. Results highlight key challenges, including species diversity, morphological overlap, and specialized domain demands, underscoring the difficulty of marine understanding. ORCA thus establishes a comprehensive benchmark to advance research in marine domain. Project Page: http://orca.hkustvgd.com/.

ORCA: Object Recognition and Comprehension for Archiving Marine Species

TL;DR

ORCA addresses the lack of standardized, domain-specific benchmarks for marine vision by assembling a large-scale multimodal dataset with 478 species, 14,647 images, 42,217 bounding boxes, and 22,321 expert-verified instance captions. It enables three core tasks—object detection (closed-set and open-vocabulary), instance captioning, and visual grounding—and introduces taxonomy-aware evaluation settings to address morphological overlap. A broad benchmark of 18 state-of-the-art models reveals that domain-specific supervision and dense, instance-level captions substantially improve localization and description, while generic vision-language models struggle to generate marine-specific, region-level outputs. ORCA thus provides a concrete, ecologically relevant platform to advance marine visual understanding and archiving, with practical implications for biodiversity monitoring and conservation through more accurate identification and documentation of marine species.

Abstract

Marine visual understanding is essential for monitoring and protecting marine ecosystems, enabling automatic and scalable biological surveys. However, progress is hindered by limited training data and the lack of a systematic task formulation that aligns domain-specific marine challenges with well-defined computer vision tasks, thereby limiting effective model application. To address this gap, we present ORCA, a multi-modal benchmark for marine research comprising 14,647 images from 478 species, with 42,217 bounding box annotations and 22,321 expert-verified instance captions. The dataset provides fine-grained visual and textual annotations that capture morphology-oriented attributes across diverse marine species. To catalyze methodological advances, we evaluate 18 state-of-the-art models on three tasks: object detection (closed-set and open-vocabulary), instance captioning, and visual grounding. Results highlight key challenges, including species diversity, morphological overlap, and specialized domain demands, underscoring the difficulty of marine understanding. ORCA thus establishes a comprehensive benchmark to advance research in marine domain. Project Page: http://orca.hkustvgd.com/.
Paper Structure (11 sections, 5 figures, 4 tables)

This paper contains 11 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Orca Construction
  4. Dataset Construction
  5. Dataset Statistic and Comparison
  6. Experiments
  7. Object Detection
  8. Instance Captioning
  9. Visual Grounding
  10. Discussion and Conclusion
  11. Acknowledgement

Figures (5)

  • Figure 1: Overview of ORCA construction process. It begins with image collection, followed by bounding box annotation and caption generation for each box. Domain experts then verify all of them and refine at least one caption per image.
  • Figure 2: ORCA offers a balanced and sufficient amount of visual and textual annotations, compared to general and domain-specific datasets.
  • Figure 3: Caption tokens length for general datasets and domain-specific datasets. The white circle represents the mean caption length. Outliers have been filtered out.
  • Figure 4: t-SNE of the vocabulary used in general datasets and domain-specific datasets). Pascal Sent. stand for Pascal Sentence for better visualization.
  • Figure 5: Quantitative results of open-vocabulary object detection, visual grounding, and image captioning.