Unlocking Zero-Shot Plant Segmentation with Pl@ntNet Intelligence

TL;DR

This paper addresses annotation bottlenecks in plant image segmentation by fusing Plantnet's plant-centric representations with the Segment Anything Model for zero-shot segmentation. The method uses PCA on Plantnet token features to generate box prompts from the first principal component and refines masks with SAM, without retraining. Across four diverse datasets, the Plantnet-DinoV2 pipeline achieves consistent $IoU$ gains over a DinoV2 baseline, demonstrating robustness to domain shifts and potential to reduce labeling effort in agricultural imaging. The results highlight the value of domain-specific foundation-model features for practical segmentation in varied field conditions.

Abstract

We present a zero-shot segmentation approach for agricultural imagery that leverages Plantnet, a large-scale plant classification model, in conjunction with its DinoV2 backbone and the Segment Anything Model (SAM). Rather than collecting and annotating new datasets, our method exploits Plantnet's specialized plant representations to identify plant regions and produce coarse segmentation masks. These masks are then refined by SAM to yield detailed segmentations. We evaluate on four publicly available datasets of various complexity in terms of contrast including some where the limited size of the training data and complex field conditions often hinder purely supervised methods. Our results show consistent performance gains when using Plantnet-fine-tuned DinoV2 over the base DinoV2 model, as measured by the Jaccard Index (IoU). These findings highlight the potential of combining foundation models with specialized plant-centric models to alleviate the annotation bottleneck and enable effective segmentation in diverse agricultural scenarios.

Figures (3)

• Figure 1: Method Pipeline, the image is patched and goes through plantnet, then each patch is classified creating a rough mask that is then turned into a box prompt and refined using SAM.
• Figure 2: Analysis of the first PCA components of the output tokens from the Plantnet model on the four datasets. We observe a clear separation between positive and negative tokens. On Phenobench, 0 is clearly a local minimum. On the other datasets, although 0 is not a local minimum, it can still serve as an effective threshold for separating the tokens into two clusters.
• Figure 3: Evolution of U-Net Model Performance with increasing training data on 4 datasets. U-Net starts to outperform our method between 20 and 40 training samples for all datasets except the Apple Tree dataset where our method is always better. For each datasets size we did 100 differents training runs and computed the mean IoU on the validation datasets which are plotted in orange points, the blue envelope being the standards deviation of the models performances.