Rethinking Plant Disease Diagnosis: Bridging the Academic-Practical Gap with Vision Transformers and Zero-Shot Learning

TL;DR

The study confronts the gap between laboratory plant-disease benchmarks and real-field images by systematically comparing CNNs, Vision Transformers, and CLIP-based zero-shot models. Trained on PlantVillage, CNNs and Transformers are evaluated on a diverse 945-image field test set, while CLIP models operate zero-shot using descriptive prompts. Transformers and especially CLIP-based zero-shot models show superior generalization and robustness to real-world conditions, with CLIP also providing natural-language interpretability that aids trust and validation. The work demonstrates zero-shot learning as a practical, scalable domain-adaptation strategy for crop health diagnostics and highlights the need for realistic data and efficient models for in-field deployment.

Abstract

Recent advances in deep learning have enabled significant progress in plant disease classification using leaf images. Much of the existing research in this field has relied on the PlantVillage dataset, which consists of well-centered plant images captured against uniform, uncluttered backgrounds. Although models trained on this dataset achieve high accuracy, they often fail to generalize to real-world field images, such as those submitted by farmers to plant diagnostic systems. This has created a significant gap between published studies and practical application requirements, highlighting the necessity of investigating and addressing this issue. In this study, we investigate whether attention-based architectures and zero-shot learning approaches can bridge the gap between curated academic datasets and real-world agricultural conditions in plant disease classification. We evaluate three model categories: Convolutional Neural Networks (CNNs), Vision Transformers, and Contrastive Language-Image Pre-training (CLIP)-based zero-shot models. While CNNs exhibit limited robustness under domain shift, Vision Transformers demonstrate stronger generalization by capturing global contextual features. Most notably, CLIP models classify diseases directly from natural language descriptions without any task-specific training, offering strong adaptability and interpretability. These findings highlight the potential of zero-shot learning as a practical and scalable domain adaptation strategy for plant health diagnosis in diverse field environments.

Figures (8)

• Figure 1: Model Categorization and Evaluation Pipeline for Plant Disease Classification
• Figure 2: Model Categorization and Evaluation Pipeline for Plant Disease Classification
• Figure 3: Sample images from the Training Dataset
• Figure 8: F1-Score Distribution Across 5 Folds for Model Performance Comparison
• Figure 9: Comparison of Confusion Matrices Across Models