Domain-Specific Self-Supervised Pre-training for Agricultural Disease Classification: A Hierarchical Vision Transformer Study

TL;DR

This work tackles agricultural disease classification under limited labeled data by evaluating domain-specific self-supervised pre-training with a hierarchical vision-transformer (HierarchicalViT). Through comprehensive experiments across three datasets, it demonstrates that domain SSL provides larger accuracy gains than architectural improvements and that the SSL benefits transfer across backbones. A key finding is that domain-specific SSL with as few as 3,000 unlabeled images can outperform large-scale ImageNet pre-training, and the resulting models are well-calibrated for deployment, especially after temperature scaling. The study offers practical guidance for data collection prioritization and deployment reliability in agricultural AI, emphasizing empirical rather than architectural novelty.

Abstract

We investigate the impact of domain-specific self-supervised pre-training on agricultural disease classification using hierarchical vision transformers. Our key finding is that SimCLR pre-training on just 3,000 unlabeled agricultural images provides a +4.57% accuracy improvement--exceeding the +3.70% gain from hierarchical architecture design. Critically, we show this SSL benefit is architecture-agnostic: applying the same pre-training to Swin-Base yields +4.08%, to ViT-Base +4.20%, confirming practitioners should prioritize domain data collection over architectural choices. Using HierarchicalViT (HVT), a Swin-style hierarchical transformer, we evaluate on three datasets: Cotton Leaf Disease (7 classes, 90.24%), PlantVillage (38 classes, 96.3%), and PlantDoc (27 classes, 87.1%). At matched parameter counts, HVT-Base (78M) achieves 88.91% vs. Swin-Base (88M) at 87.23%, a +1.68% improvement. For deployment reliability, we report calibration analysis showing HVT achieves 3.56% ECE (1.52% after temperature scaling). Code: https://github.com/w2sg-arnav/HierarchicalViT

Figures (8)

• Figure 1: HierarchicalViT-XL architecture overview. The model processes $448 \times 448$ RGB images through patch embedding and four hierarchical transformer stages with progressive spatial downsampling ($32^2 \to 16^2 \to 8^2 \to 4^2$ tokens) and channel expansion ($192 \to 384 \to 768 \to 1536$ dimensions). Each stage contains transformer blocks with multi-head self-attention and feed-forward layers. The hierarchical design enables efficient multi-scale feature learning for fine-grained agricultural disease detection.
• Figure 2: Confusion matrix on the test set showing classification patterns across all 7 disease classes: Bacterial Blight, Curl Virus, Fusarium Wilt, Grey Mildew, Healthy Leaf, Leaf Reddening, and Target Spot. Most predictions are concentrated along the diagonal. Common misclassifications occur between visually similar classes (Grey Mildew vs. Target Spot, Bacterial Blight vs. Fusarium Wilt).
• Figure 3: Attention rollout visualization on representative cotton leaf disease samples. Top row shows original images from four of the seven disease classes. Bottom row displays attention heatmaps from HVT's final stage, showing focus on discriminative regions (lesions, discoloration patterns). Note: Visualization images may show subset of classes or use augmented examples for clarity.
• Figure 4: Qualitative comparison of HVT against baseline approaches. (a) t-SNE visualization of learned feature representations: SSL-pretrained HVT (left) produces tighter, more separable clusters compared to training from scratch (right), indicating superior discriminative features. Different colors represent the seven cotton disease classes. (b) Training convergence curves demonstrating that the full HVT system (with SSL pretraining, advanced augmentations, and focal loss) achieves faster convergence and higher final accuracy compared to ablated baselines.
• Figure 5: Training and validation curves over 100 epochs.