AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding

Abstract

The deployment of Multimodal Large Language Models (MLLMs) in agriculture is currently stalled by a critical trade-off: the existing literature lacks the large-scale agricultural datasets required for robust model development and evaluation, while current state-of-the-art models lack the verified domain expertise necessary to reason across diverse taxonomies. To address these challenges, we propose the Vision-to-Verified-Knowledge (V2VK) pipeline, a novel generative AI-driven annotation framework that integrates visual captioning with web-augmented scientific retrieval to autonomously generate the AgriMM benchmark, effectively eliminating biological hallucinations by grounding training data in verified phytopathological literature. The AgriMM benchmark contains over 3,000 agricultural classes and more than 607k VQAs spanning multiple tasks, including fine-grained plant species identification, plant disease symptom recognition, crop counting, and ripeness assessment. Leveraging this verifiable data, we present AgriChat, a specialized MLLM that presents broad knowledge across thousands of agricultural classes and provides detailed agricultural assessments with extensive explanations. Extensive evaluation across diverse tasks, datasets, and evaluation conditions reveals both the capabilities and limitations of current agricultural MLLMs, while demonstrating AgriChat's superior performance over other open-source models, including internal and external benchmarks. The results validate that preserving visual detail combined with web-verified knowledge constitutes a reliable pathway toward robust and trustworthy agricultural AI. The code and dataset are publicly available at https://github.com/boudiafA/AgriChat .

Figures (11)

• Figure 1: Illustrative examples of AgriChat's conversational diagnostic capabilities across three core agricultural tasks. Left: Fine-grained plant species identification with follow-up knowledge queries. Center: Plant disease symptom recognition, correctly diagnosing Sigatoka disease from visible leaf lesions. Right: Fruit counting and ripeness assessment from a single field image. These examples highlight AgriChat's ability to support interactive, expert-level agricultural reasoning beyond simple classification.
• Figure 2: (a) Comparison between existing foundation models (Llama-3.2 llama3-2024, LLava-OneVision llava-onevision2024, Qwen-2.5 qwen2024) and our proposed agriculture-focused AgriChat model which achieved superior performance across four agriculture benchmarks. In comparison to existing agriculture benchmark datasets (PlantVillageVQA plantvillagevqa2024, AGMMU AGMMU), our proposed large-scale benchmark AgriMM exceeds the state-of-the-art benchmarks in terms of (b) number of images, (c) number of classes, and (d) number of VQA pairs.
• Figure 3: AgriMM dataset statistics showing (a) taxonomic hierarchy and (b) class balance across functional categories.
• Figure 4: Overview of the Vision-to-Verified-Knowledge Synthesis Pipeline. Visual features are grounded in verified scientific literature through a three-stage generation and verification process.
• Figure 5: Overview of the AgriChat Architecture. The model utilizes an adaptive resolution strategy to process high-quality agricultural imagery. Input images are split into local patches and a resized global thumbnail, encoded by SigLIP equipped with LoRA adapters, and aligned via a projection layer. The visual tokens are concatenated with text instructions and processed by LLM backbone equipped with LoRA adapters (SwiGLU + Multi-Head Self-Attention) to generate verifiable diagnostic responses.