Automated Marine Biofouling Assessment: Benchmarking Computer Vision and Multimodal LLMs on the Level of Fouling Scale

TL;DR

The paper tackles the problem of scalable, objective marine biofouling assessment across LoF levels $0$–$5$. It benchmarks convolutional neural networks, transformer-based segmentation, and zero-shot multimodal LLM pipelines on a shared expert-labelled MPI dataset to classify LoF with varying degrees of granularity. CNNs excel at the LoF extremes but lack interpretability, SegFormer provides interpretable coverage maps yet struggles with mixed-state fouling, and LLMs offer contextual reasoning and interpretability in zero-shot settings when prompted effectively, with RAG grounding further enhancing alignment. The results indicate complementary strengths among approaches and motivate hybrid systems that combine segmentation-derived coverage with LLM reasoning to enable scalable, interpretable biofouling assessment suitable for marine biosecurity workflows.

Abstract

Marine biofouling on vessel hulls poses major ecological, economic, and biosecurity risks. Traditional survey methods rely on diver inspections, which are hazardous and limited in scalability. This work investigates automated classification of biofouling severity on the Level of Fouling (LoF) scale using both custom computer vision models and large multimodal language models (LLMs). Convolutional neural networks, transformer-based segmentation, and zero-shot LLMs were evaluated on an expert-labelled dataset from the New Zealand Ministry for Primary Industries. Computer vision models showed high accuracy at extreme LoF categories but struggled with intermediate levels due to dataset imbalance and image framing. LLMs, guided by structured prompts and retrieval, achieved competitive performance without training and provided interpretable outputs. The results demonstrate complementary strengths across approaches and suggest that hybrid methods integrating segmentation coverage with LLM reasoning offer a promising pathway toward scalable and interpretable biofouling assessment.

Figures (8)

• Figure 1: Examples of propeller biofouling across the six Level of Fouling (LoF) categories (0–5). Each subfigure illustrates the progressive increase in fouling severity as defined by the LoF scale.
• Figure 2: Decision tree for determining the level of fouling of a ship given the observations of the hull.
• Figure 3: Example of an image with class labels, where Macrofouling is purple, Slime is green, and Clean is yellow.
• Figure 4: Each column represents the number of images in each respective class, where the colour of the column represents the predicted LoF rating. The left side represents the most confident prediction, while the right side represents the second most confident prediction.
• Figure 5: Different output predictions across two consecutive frames of video inference showing the instability in the micro- and macro-fouling predictions.