Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agro-Consensus: Semantic Self-Consistency in Vision-Language Models for Crop Disease Management in Developing Countries

Mihir Gupta, Pratik Desai, Ross Greer

TL;DR

This work tackles the challenge of reliable crop-disease captioning in resource-constrained settings by introducing a semantic self-consistency framework for vision-language models. It combines a lightweight embedding-based clustering with cosine-similarity voting and a HITL crop-confirmation step to produce coherent, expert-like captions without cloud-dependence. Applied to PlantVillage with a fine-tuned $3$-b PaliGemma model, the approach yields a peak single-cluster accuracy of $83.1\%$ and up to $94.0\%$ accuracy when selecting among top $4$ clusters, outperforming greedy decoding baselines. The method emphasizes on-device feasibility, requiring modest resources (e.g., an embedding model of about $80$ MB) and a practical evaluation protocol using a prompt-based plant-pathologist scorer.

Abstract

Agricultural disease management in developing countries such as India, Kenya, and Nigeria faces significant challenges due to limited access to expert plant pathologists, unreliable internet connectivity, and cost constraints that hinder the deployment of large-scale AI systems. This work introduces a cost-effective self-consistency framework to improve vision-language model (VLM) reliability for agricultural image captioning. The proposed method employs semantic clustering, using a lightweight (80MB) pre-trained embedding model to group multiple candidate responses. It then selects the most coherent caption -- containing a diagnosis, symptoms, analysis, treatment, and prevention recommendations -- through a cosine similarity-based consensus. A practical human-in-the-loop (HITL) component is incorporated, wherein user confirmation of the crop type filters erroneous generations, ensuring higher-quality input for the consensus mechanism. Applied to the publicly available PlantVillage dataset using a fine-tuned 3B-parameter PaliGemma model, our framework demonstrates improvements over standard decoding methods. Evaluated on 800 crop disease images with up to 21 generations per image, our single-cluster consensus method achieves a peak accuracy of 83.1% with 10 candidate generations, compared to the 77.5% baseline accuracy of greedy decoding. The framework's effectiveness is further demonstrated when considering multiple clusters; accuracy rises to 94.0% when a correct response is found within any of the top four candidate clusters, outperforming the 88.5% achieved by a top-4 selection from the baseline.

Agro-Consensus: Semantic Self-Consistency in Vision-Language Models for Crop Disease Management in Developing Countries

TL;DR

This work tackles the challenge of reliable crop-disease captioning in resource-constrained settings by introducing a semantic self-consistency framework for vision-language models. It combines a lightweight embedding-based clustering with cosine-similarity voting and a HITL crop-confirmation step to produce coherent, expert-like captions without cloud-dependence. Applied to PlantVillage with a fine-tuned -b PaliGemma model, the approach yields a peak single-cluster accuracy of and up to accuracy when selecting among top clusters, outperforming greedy decoding baselines. The method emphasizes on-device feasibility, requiring modest resources (e.g., an embedding model of about MB) and a practical evaluation protocol using a prompt-based plant-pathologist scorer.

Abstract

Agricultural disease management in developing countries such as India, Kenya, and Nigeria faces significant challenges due to limited access to expert plant pathologists, unreliable internet connectivity, and cost constraints that hinder the deployment of large-scale AI systems. This work introduces a cost-effective self-consistency framework to improve vision-language model (VLM) reliability for agricultural image captioning. The proposed method employs semantic clustering, using a lightweight (80MB) pre-trained embedding model to group multiple candidate responses. It then selects the most coherent caption -- containing a diagnosis, symptoms, analysis, treatment, and prevention recommendations -- through a cosine similarity-based consensus. A practical human-in-the-loop (HITL) component is incorporated, wherein user confirmation of the crop type filters erroneous generations, ensuring higher-quality input for the consensus mechanism. Applied to the publicly available PlantVillage dataset using a fine-tuned 3B-parameter PaliGemma model, our framework demonstrates improvements over standard decoding methods. Evaluated on 800 crop disease images with up to 21 generations per image, our single-cluster consensus method achieves a peak accuracy of 83.1% with 10 candidate generations, compared to the 77.5% baseline accuracy of greedy decoding. The framework's effectiveness is further demonstrated when considering multiple clusters; accuracy rises to 94.0% when a correct response is found within any of the top four candidate clusters, outperforming the 88.5% achieved by a top-4 selection from the baseline.
Paper Structure (13 sections, 2 figures, 1 table)

This paper contains 13 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset Preparation and Caption Generation
  5. Vision-Language Model Fine-tuning
  6. Multi-Response Generation for Evaluation
  7. Agricultural Image Processing Pipeline and Cosine-Consistency Voting
  8. Experimental Setup
  9. Dataset Partitioning
  10. Prompt-Based Scoring
  11. Evaluation Methodology
  12. Results and Analysis
  13. Discussion and Conclusions

Figures (2)

  • Figure 1: Farmer-captured crop images are analyzed by a fine-tuned vision-language model that produces candidate diagnoses; texts are encoded using a pre-trained embedding model, and cosine-consistency voting selects the most coherent diagnosis.
  • Figure 2: Image of a tomato leaf from PlantVillage dataset with its synthetic caption: "tomato early blight, infection level is moderate to severe; symptoms: dark brown circular spots, concentric rings, prominent yellowing, necrosis; analysis: established infection requires immediate fungicidal application; fungicides: chlorothalonil, mancozeb, azoxystrobin; prevention: crop rotation, resistant varieties, proper spacing, sanitation"