Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2

TL;DR

This study rigorously compares zero-shot and prompt-based segmentation for solar PV cells in RGB aerial imagery using SAM 2, SAM 2.1, and a CNN baseline (Eff-UNet) across multiple resolutions and lighting conditions. It evaluates prompt strategies, including user boxes, user points, and YOLO-generated boxes, and shows that SAM 2.1 with user-box prompts provides robust performance across conditions, while high-resolution, optimally-lit data still favor CNN-based automatic segmentation. The results highlight a trade-off between accuracy and compute: SAM-based workflows offer flexibility and reduced manual annotation in challenging scenes but are more resource-intensive than CNNs; YOLO-prompted SAMs can mitigate some input quality issues. The findings guide practitioners on when to deploy prompt-based SAMs and emphasize SAM 2.1’s potential to decrease annotation workload in RS PV-cell segmentation, with future work extending analysis to other RS sensors and reducing prompting requirements.

Abstract

This paper provides insights on the effectiveness of the zero shot, prompt-based Segment Anything Model (SAM) and its updated versions, SAM 2 and SAM 2.1, along with the non-promptable conventional neural network (CNN), for segmenting solar panels in RGB aerial remote sensing imagery. The study evaluates these models across diverse lighting conditions, spatial resolutions, and prompt strategies. SAM 2 showed slight improvements over SAM, while SAM 2.1 demonstrated notable improvements, particularly in sub-optimal lighting and low resolution conditions. SAM models, when prompted by user-defined boxes, outperformed CNN in all scenarios; in particular, user-box prompts were found crucial for achieving reasonable performance in low resolution data. Additionally, under high resolution, YOLOv9 automatic prompting outperformed user-points prompting by providing reliable prompts to SAM. Under low resolution, SAM 2.1 prompted by user points showed similar performance to SAM 2.1 prompted by YOLOv9, highlighting its zero shot improvements with a single click. In high resolution with optimal lighting imagery, Eff-UNet outperformed SAMs prompted by YOLOv9, while under sub-optimal lighting conditions, Eff-UNet, and SAM 2.1 prompted by YOLOv9, had similar performance. However, SAM is more resource-intensive, and despite improved inference time of SAM 2.1, Eff-UNet is more suitable for automatic segmentation in high resolution data. This research details strengths and limitations of each model and outlines the robustness of user-prompted image segmentation models.

Figures (6)

• Figure 1: PV cells captured in images with different resolutions and lighting conditions: High resolution (0.15 m) and low resolution image (0.25 m), highlighting the increased difficulty in segmentation as resolution decreases and lighting condition becomes sub-optimal.
• Figure 2: Upper chart: IoU and F1-score (underlined) results across all three studied datasets. Lower chart: mean IoU and F1-score across the datasets.
• Figure 3: SAM Vs. SAM 2: Under sub-optimal lighting imagery, SAM 2 segments PVCs in full while SAM segments them partially, or falsely (Green: TP, Red: FP & FN).
• Figure 4: SAM 2 Vs. SAM 2.1 under low resolution: Shadow are clustered with small PVCs segmented by SAM 2 as PVCs (Red: FP & FN) whereas SAM 2.1 segments correctly.
• Figure 5: Prompts: under sub-optimal lighting conditions YOLO exceeds single click prompts. boxes lowering FPR by defining the background regions. (Green: TP, Red: FP & FN).