Designing A Sustainable Marine Debris Clean-up Framework without Human Labels

TL;DR

The paper presents a sustainable, label-free framework for marine debris detection, classification, and mapping from drone imagery. It combines Grounding DINO for zero-shot object detection with CLIP for zero-shot classification across seven debris-material classes, plus a SIFT-based duplicate removal and a Ground Sample Distance-based map visualization to support cleanup planning. Evaluated on a novel Allen Island dataset (1,000 images; 200 labeled), the approach achieves a mean IoU of 0.69 for detection and a ~0.74 F1 score (≈0.775 accuracy) for classification without task-specific labels, on par with supervised methods. A user-friendly web interface and workflow enable community stakeholders to coordinate cleanup efforts, demonstrating practical impact and avenues for broader deployment and future refinements.

Abstract

Marine debris poses a significant ecological threat to birds, fish, and other animal life. Traditional methods for assessing debris accumulation involve labor-intensive and costly manual surveys. This study introduces a framework that utilizes aerial imagery captured by drones to conduct remote trash surveys. Leveraging computer vision techniques, our approach detects, classifies, and maps marine debris distributions. The framework uses Grounding DINO, a transformer-based zero-shot object detector, and CLIP, a vision-language model for zero-shot object classification, enabling the detection and classification of debris objects based on material type without the need for training labels. To mitigate over-counting due to different views of the same object, Scale-Invariant Feature Transform (SIFT) is employed for duplicate matching using local object features. Additionally, we have developed a user-friendly web application that facilitates end-to-end analysis of drone images, including object detection, classification, and visualization on a map to support cleanup efforts. Our method achieves competitive performance in detection (0.69 mean IoU) and classification (0.74 F1 score) across seven debris object classes without labeled data, comparable to state-of-the-art supervised methods. This framework has the potential to streamline automated trash sampling surveys, fostering efficient and sustainable community-led cleanup initiatives.

Figures (6)

• Figure 1: Beverage bottle, fishing gear and other marine debris along a beach on Allen Island in Maine, USA where the data was collected.
• Figure 2: System diagram of the proposed framework.
• Figure 3: Examples of detections obtained from the raw query "all trashes." Detections are highlighted using red bounding boxes to indicate the identified areas within the dataset.
• Figure 4: Example debris object from each of seven classes.
• Figure 5: Detected clusters of marine debris distribution.