Zero-Shot Segmentation through Prototype-Guidance for Multi-Label Plant Species Identification

TL;DR

This work tackles fine-grained multi-label plant species identification in high-resolution vegetation plots under domain shift by introducing a segmentation-first approach. A narrow Vision Transformer is trained on a proxy objective to reconstruct $7806$ prototypes derived from $K$-means clustering on DINOv2 embeddings, using test-image crops processed through a frozen DINOv2 backbone to generate attention-based localization. Inference leverages averaged attention maps to select regions and employs two heuristics—patch-wise and grid-assembly—paired with a thresholded classifier, with the grid strategy delivering the best $F_1$ performance ($\approx 0.333$). The method achieved 5th place on PlantCLEF 2025 private leaderboard, demonstrating that context-rich, region-focused representations are crucial for robust multi-label plant identification in complex ecological imagery, and offering a promising direction for domain-adaptive, weakly supervised segmentation approaches.

Abstract

This paper presents an approach developed to address the PlantClef 2025 challenge, which consists of a fine-grained multi-label species identification, over high-resolution images. Our solution focused on employing class prototypes obtained from the training dataset as a proxy guidance for training a segmentation Vision Transformer (ViT) on the test set images. To obtain these representations, the proposed method extracts features from training dataset images and create clusters, by applying K-Means, with $K$ equals to the number of classes in the dataset. The segmentation model is a customized narrow ViT, built by replacing the patch embedding layer with a frozen DinoV2, pre-trained on the training dataset for individual species classification. This model is trained to reconstruct the class prototypes of the training dataset from the test dataset images. We then use this model to obtain attention scores that enable to identify and localize areas of interest and consequently guide the classification process. The proposed approach enabled a domain-adaptation from multi-class identification with individual species, into multi-label classification from high-resolution vegetation plots. Our method achieved fifth place in the PlantCLEF 2025 challenge on the private leaderboard, with an F1 score of 0.33331. Besides that, in absolute terms our method scored 0.03 lower than the top-performing submission, suggesting that it may achieved competitive performance in the benchmark task. Our code is available at \href{https://github.com/ADAM-UEFS/PlantCLEF2025}{https://github.com/ADAM-UEFS/PlantCLEF2025}.

Figures (5)

• Figure 1: Training process diagram.
• Figure 2: Inference stage diagram.
• Figure 3: Qualitative results from the segmentation ViT during an early training epoch. The model correctly assigns high attention scores (red) to plant-containing regions and low scores (blue) to background elements, including the quadrat frame.
• Figure 4: Illustration of attention map degradation over several training epochs. While initial epochs exhibit focused attention on plant regions, later epochs show a diffusion of attention and an incorrect intensification on background areas.
• Figure 5: Training loss curve over epochs. The onset of loss convergence (plateau) aligns with the observed degradation in attention map quality, suggesting the model began to overfit to the reconstruction task.