A Vision-Language Foundation Model for Leaf Disease Identification

TL;DR

SCOLD introduces a context-aware soft-target contrastive learning framework to train a vision-language foundation model for leaf disease identification, addressing domain-specific image-text alignment with descriptive symptom text. Leveraging LeafNet—a large-scale dataset with >186k images and 97 concepts—SCOLD achieves strong zero-shot and few-shot performance and competitive image-text retrieval while maintaining efficiency. Across ten leaf-disease benchmarks, it outperforms OpenAI-CLIP-L, BioCLIP, and SigLIP2, with ablations confirming the benefits of long-context prompts and soft-targets. This work provides a practical foundation for multimodal plant-disease diagnostics in smart agriculture and releases code on HuggingFace for further research and deployment.

Abstract

Leaf disease identification plays a pivotal role in smart agriculture. However, many existing studies still struggle to integrate image and textual modalities to compensate for each other's limitations. Furthermore, many of these approaches rely on pretraining with constrained datasets such as ImageNet, which lack domain-specific information. We propose SCOLD (Soft-target COntrastive learning for Leaf Disease identification), a context-aware vision-language foundation model tailored to address these challenges for agricultural tasks. SCOLD is developed using a diverse corpus of plant leaf images and corresponding symptom descriptions, comprising over 186,000 image-caption pairs aligned with 97 unique concepts. Through task-agnostic pretraining, SCOLD leverages contextual soft targets to mitigate overconfidence in contrastive learning by smoothing labels, thereby improving model generalization and robustness on fine-grained classification tasks. Experimental results demonstrate that SCOLD outperforms existing vision-language models such as OpenAI-CLIP-L, BioCLIP, and SigLIP2 across several benchmarks, including zero-shot and few-shot classification, image-text retrieval, and image classification, while maintaining a competitive parameter footprint. Ablation studies further highlight SCOLD's effectiveness in contrast to its counterparts. The proposed approach significantly advances the agricultural vision-language foundation model, offering strong performance with minimal or no supervised fine-tuning. This work lays a solid groundwork for future research on models trained with long-form and simplified contexts, tasks involving class ambiguity, and multi-modal systems for intelligent plant disease diagnostics. The code for this study is available at https://huggingface.co/enalis/scold

Figures (11)

• Figure 1: The overall framework of .
• Figure 2: The unified model training in in (c) as Image-Text Alignment with $\mathcal{H}(\bullet )$ as a similarity matrix (white colour) of image-to-text or text-to-image. The limitation problem is that (a) Image classification and (b) Image-Text Classification using $\mathcal{H}(\bullet )$ is constant for each model(red colour).
• Figure 3: A detailed illustration of image-text alignment with a prompt template for enriching class information with a common plant-leaf disease dataset such as PlantVillage ref29.
• Figure 4: Disentangling the negatives in similar distributions. With an image query Tomato Leaf Mould, the probability distribution of the hard label can tend to appear in different crop classes, such as Potato or Wheat, due to distance-only optimization. Meanwhile, with , this probability distribution will be distributed into the most similar classes to the query.
• Figure 5: Illustrative comparisons across different learning paradigms. With a batch of size $|B|$, all visual features $v$ and textual features $t$ are in the same dimension, and $C$ is the number of classes. Given a similarity matrix for each method, the labels are defined as the positive pairs whose elements are purple and whose negatives are white. (c) ref24 or ref28 has a one-to-one assumptions for each pair. Meanwhile, (a) has many-to-many assumptions based on the context; positive pairs are considered based on orange, related pairs are negative yellow, and the remaining non-related pairs are white.