Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Lightweight and Explainable Vision-Language Framework for Crop Disease Visual Question Answering

Md. Zahid Hossain, Most. Sharmin Sultana Samu, Md. Rakibul Islam, Md. Siam Ansary

TL;DR

The paper addresses the need for practical visual question answering in crop disease analysis by introducing a lightweight, explainable vision–language framework that combines a Swin Transformer vision encoder with sequence-to-sequence decoders. It employs a two-stage training regime to separately optimize visual representation learning and cross-modal reasoning, achieving high plant and disease identification accuracy while delivering strong natural language generation performance with fewer parameters than large baselines. Explainability is demonstrated via Grad-CAM and token-level attribution, supporting transparent vision–language reasoning. The approach shows promise for real-world agricultural deployment, with robust performance across diverse user queries and efficient inference, paving the way for broader task-specific, data-efficient VQA in plant health management.

Abstract

Visual question answering for crop disease analysis requires accurate visual understanding and reliable language generation. This work presents a lightweight vision-language framework for crop and disease identification from leaf images. The proposed approach combines a Swin Transformer vision encoder with sequence-to-sequence language decoders. A two-stage training strategy is adopted to improve visual representation learning and cross-modal alignment. The model is evaluated on a large-scale crop disease dataset using classification and natural language generation metrics. Experimental results show high accuracy for both crop and disease identification. The framework also achieves strong performance on BLEU, ROUGE and BERTScore. Our proposed models outperform large-scale vision-language baselines while using significantly fewer parameters. Explainability is assessed using Grad-CAM and token-level attribution. Qualitative results demonstrate robust performance under diverse user-driven queries. These findings highlight the effectiveness of task-specific visual pretraining for crop disease visual question answering.

A Lightweight and Explainable Vision-Language Framework for Crop Disease Visual Question Answering

TL;DR

The paper addresses the need for practical visual question answering in crop disease analysis by introducing a lightweight, explainable vision–language framework that combines a Swin Transformer vision encoder with sequence-to-sequence decoders. It employs a two-stage training regime to separately optimize visual representation learning and cross-modal reasoning, achieving high plant and disease identification accuracy while delivering strong natural language generation performance with fewer parameters than large baselines. Explainability is demonstrated via Grad-CAM and token-level attribution, supporting transparent vision–language reasoning. The approach shows promise for real-world agricultural deployment, with robust performance across diverse user queries and efficient inference, paving the way for broader task-specific, data-efficient VQA in plant health management.

Abstract

Visual question answering for crop disease analysis requires accurate visual understanding and reliable language generation. This work presents a lightweight vision-language framework for crop and disease identification from leaf images. The proposed approach combines a Swin Transformer vision encoder with sequence-to-sequence language decoders. A two-stage training strategy is adopted to improve visual representation learning and cross-modal alignment. The model is evaluated on a large-scale crop disease dataset using classification and natural language generation metrics. Experimental results show high accuracy for both crop and disease identification. The framework also achieves strong performance on BLEU, ROUGE and BERTScore. Our proposed models outperform large-scale vision-language baselines while using significantly fewer parameters. Explainability is assessed using Grad-CAM and token-level attribution. Qualitative results demonstrate robust performance under diverse user-driven queries. These findings highlight the effectiveness of task-specific visual pretraining for crop disease visual question answering.
Paper Structure (21 sections, 12 figures, 9 tables)

This paper contains 21 sections, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Question Answering Frameworks for Agricultural Disease Diagnosis
  4. Multimodal Deep Learning and Transformer-Based Models
  5. Knowledge-Enhanced and Large Language Model–Driven Agricultural Assistants
  6. Datasets, Benchmarking and Task-Specific Learning Strategies
  7. Methodology
  8. Dataset
  9. Proposed Methodology
  10. Evaluation Metrics
  11. Experimental Setup
  12. Result Analysis
  13. Plant and Disease Identification Performance
  14. Natural Language Generation Performance
  15. Semantic Similarity Evaluation
  16. ...and 6 more sections

Figures (12)

  • Figure 1: Distribution of plant categories by number of images.
  • Figure 2: Distribution of disease categories by number of images.
  • Figure 3: Distribution of plant–disease combinations by number of images.
  • Figure 4: Two-stage architecture of the proposed framework. Stage 1 learns plant and disease representations using a shared Swin-T encoder, while Stage 2 reuses the frozen encoder for visual question answering with a stacked text decoder.
  • Figure 5: Grad-CAM visualization highlighting diseased regions in an apple leaf image using Swin--T5.
  • ...and 7 more figures