Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models

Abstract

The detection and classification of bacterial colonies in images of agar-plates is important in microbiology, but is hindered by the lack of labeled datasets. Therefore, we propose Colony Grounded SAM2, a zero-shot inference pipeline to detect and segment bacterial colonies in multiple settings without any further training. By utilizing the pre-trained foundation models Grounding DINO and Segment Anything Model 2, fine-tuned to the microbiological domain, we developed a model that is robust to data changes. Results showed a mean Average Precision of 93.1\% and a $Dice@detection$ score of 0.85, showing excellent detection and segmentation capabilities on out-of-distribution datasets. The entire pipeline with model weights are shared open access to aid with annotation- and classification purposes in microbiology.

Figures (3)

• Figure 1: The proposed Colony Grounded SAM2 pipeline for bacterial colonies
• Figure 2: Qualitative detection results illustrating model behavior across various conditions. Bacterial colony types are labeled by their ID as defined in Table 1 from the ADBC papermakrai2023annotated. Both ground truths (GT) and predictions are shown. Predictions correctly matched to ground truths are green, unmatched ground truths are yellow (FN = False Negative), and unmatched predictions are red (FP = False Positive).
• Figure 3: Qualitative segmentation results on the Hemolysis dataset. Correctly matched colonies are shown as green, unmatched ground truths are yellow, and unmatched predictions are red.