FathomVerse: A community science dataset for ocean animal discovery

TL;DR

FathomVerse addresses the challenge of deep-sea animal discovery by combining a community-science game with expert-backed consensus labeling to build FathomVerse v0, a 3843-image dataset with 8092 bounding boxes across 12 morph groups from two new deep-sea locations. The approach leverages player annotations filtered by F1-score to produce reliable labels and evaluates detector performance using both FathomNet-derived and FathomVerse-specific models. Key contributions include a scalable annotation pipeline for hard-to-identify benthic fauna, analysis of player performance and annotation reliability, and baseline detectors highlighting the need for architectural advances to handle deep-sea visual variability. This work lays groundwork for improved fine-grained transfer learning, novel category discovery, and conservation-relevant analyses in ocean science, while outlining practical paths to expand categories and background contextualization in future iterations.

Abstract

Can computer vision help us explore the ocean? The ultimate challenge for computer vision is to recognize any visual phenomena, more than only the objects and animals humans encounter in their terrestrial lives. Previous datasets have explored everyday objects and fine-grained categories humans see frequently. We present the FathomVerse v0 detection dataset to push the limits of our field by exploring animals that rarely come in contact with people in the deep sea. These animals present a novel vision challenge. The FathomVerse v0 dataset consists of 3843 images with 8092 bounding boxes from 12 distinct morphological groups recorded at two locations on the deep seafloor that are new to computer vision. It features visually perplexing scenarios such as an octopus intertwined with a sea star, and confounding categories like vampire squids and sea spiders. This dataset can push forward research on topics like fine-grained transfer learning, novel category discovery, species distribution modeling, and carbon cycle analysis, all of which are important to the care and husbandry of our planet.

Figures (13)

• Figure 1: Various screens in FathomVerse v0 Game. (a) Active Missions - Players 'dive' to collect three mission animals, and learn about these animals in Mission Briefings. (b) Collection - Players experience the deep via an avatar that swims with the currents. Animals appear in bubbles that can be collected. (c) Labeling - Players select one of the three mission animal icons or None of the These to label each animal they collected while diving. (d) Verification - After players submit their annotations, feedback (e.g., correct, incorrect, missed) is shown to the player.
• Figure 2: Dive imagery. For each active animal mission group, players are shown images that are either positives, easy negatives, or hard negatives. Example images for octopus (top row) and urchin (bottom row) are shown. Hard negatives for octopus include vampire squid; hard negatives for urchin include anemones and radiolarians.
• Figure 3: Raw player annotation counts during beta testing. Left axis tracks number of annotations per animal mission group across waves A-D. Right axis tracks overall number of annotations during beta testing per animal mission group.
• Figure 4: Individual player performance. (a) Accuracy of each player compared to labels from an expert marine biologist. (b) Player F1 scores compared to labels from an expert. Colors indicate players from different release waves of the game experiment: blue dots are for the waves labeling data from Musicians Seamount; green dots are for the waves labeling Octopus garden images. The overall highest contributing player is from Wave B, and is likely an ocean enthusiast.
• Figure 5: Player performance in FathomVerse v0 dataset. (a) Precision (blue line) and recall (organge line) of the dataset as a function of F1 threshold of players. Player's reaching an F1 threshold of 0.8 are indicated by black x. (b) Precision and recall of data from players achieving F1 >= 0.8 as a function of total number of annotations.