Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE

Brendan Campbell, Alan Williams, Kleio Baxevani, Alyssa Campbell, Rushabh Dhoke, Rileigh E. Hudock, Xiaomin Lin, Vivek Mange, Bernhard Neuberger, Arjun Suresh, Alhim Vera, Arthur Trembanis, Herbert G. Tanner, Edward Hale

TL;DR

This study evaluates the ODYSSEE DL model for identifying live oysters on reefs against human annotators to assess its current reliability for non-destructive monitoring. ODYSSEE, based on YOLOv10 and trained with a mix of real and synthetic oyster imagery, processes field data orders of magnitude faster than humans but shows lower accuracy and higher false positives. Image quality strongly influences human consistency, while the model's performance declines with higher image quality, revealing gaps in training data and annotation granularity. The work highlights ODYSSEE's potential for rapid reef surveys and harvest-path planning, while outlining concrete improvements—more high-quality live imagery, expanded live/dead/unknown classes, and robust ground-truth validation—needed to make DL-based reef censusing a dependable tool for restoration and aquaculture management.

Abstract

Oysters are ecologically and commercially important species that require frequent monitoring to track population demographics (e.g. abundance, growth, mortality). Current methods of monitoring oyster reefs often require destructive sampling methods and extensive manual effort. Therefore, they are suboptimal for small-scale or sensitive environments. A recent alternative, the ODYSSEE model, was developed to use deep learning techniques to identify live oysters using video or images taken in the field of oyster reefs to assess abundance. The validity of this model in identifying live oysters on a reef was compared to expert and non-expert annotators. In addition, we identified potential sources of prediction error. Although the model can make inferences significantly faster than expert and non-expert annotators (39.6 s, $2.34 \pm 0.61$ h, $4.50 \pm 1.46$ h, respectively), the model overpredicted the number of live oysters, achieving lower accuracy (63\%) in identifying live oysters compared to experts (74\%) and non-experts (75\%) alike. Image quality was an important factor in determining the accuracy of the model and the annotators. Better quality images improved human accuracy and worsened model accuracy. Although ODYSSEE was not sufficiently accurate, we anticipate that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes will greatly improve the model's predictive power based on the results of this analysis. Future research should address methods that improve the detection of living vs. dead oysters.

Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE

TL;DR

This study evaluates the ODYSSEE DL model for identifying live oysters on reefs against human annotators to assess its current reliability for non-destructive monitoring. ODYSSEE, based on YOLOv10 and trained with a mix of real and synthetic oyster imagery, processes field data orders of magnitude faster than humans but shows lower accuracy and higher false positives. Image quality strongly influences human consistency, while the model's performance declines with higher image quality, revealing gaps in training data and annotation granularity. The work highlights ODYSSEE's potential for rapid reef surveys and harvest-path planning, while outlining concrete improvements—more high-quality live imagery, expanded live/dead/unknown classes, and robust ground-truth validation—needed to make DL-based reef censusing a dependable tool for restoration and aquaculture management.

Abstract

Oysters are ecologically and commercially important species that require frequent monitoring to track population demographics (e.g. abundance, growth, mortality). Current methods of monitoring oyster reefs often require destructive sampling methods and extensive manual effort. Therefore, they are suboptimal for small-scale or sensitive environments. A recent alternative, the ODYSSEE model, was developed to use deep learning techniques to identify live oysters using video or images taken in the field of oyster reefs to assess abundance. The validity of this model in identifying live oysters on a reef was compared to expert and non-expert annotators. In addition, we identified potential sources of prediction error. Although the model can make inferences significantly faster than expert and non-expert annotators (39.6 s, h, h, respectively), the model overpredicted the number of live oysters, achieving lower accuracy (63\%) in identifying live oysters compared to experts (74\%) and non-experts (75\%) alike. Image quality was an important factor in determining the accuracy of the model and the annotators. Better quality images improved human accuracy and worsened model accuracy. Although ODYSSEE was not sufficiently accurate, we anticipate that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes will greatly improve the model's predictive power based on the results of this analysis. Future research should address methods that improve the detection of living vs. dead oysters.
Paper Structure (8 sections, 5 figures)

This paper contains 8 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Image Acquisition
  4. Deep Learning Model
  5. Annotation Process
  6. Analysis of Model Performance
  7. Results
  8. Discussion

Figures (5)

  • Figure 1: The model appears to over-predict live (orange) oysters compared to expert and non-expert annotators. Expert annotators also make a greater number of dead (green) and unknown (purple) annotations per figure compared to non-experts. The model treatment has a wider box for live annotations since it only has the capability of making live identifications.
  • Figure 2: Examples of images used that represented low (left) and high (right) relative agreement between all annotators.
  • Figure 3: Confusion matrix from the model demonstrates that the majority of classifications were not identified by the model, primarily 'unknown' oysters. A classification or observation of '0' denotes a false positive or missed observation.
  • Figure 4: Confusion matrices from all data across expert and non-expert annotators (top left) and separated by QS show an increase in prediction accuracy and reduction in false positive detections (denoted as '0') with increasing QS.
  • Figure 5: Receiver Operating Characteristic (ROC) curve plots including the dataset where all observations and classifications were simplified to 'live' or 'not observed' for all annotator groups. Plots and line types are separated by quality score and color is separated by annotator group. Data including all QSs is represented in the top left plot. Higher area under the curve values observed with increasing QS for expert and non-expert annotators and lower area under the curve values observed for the model with increasing QS.