Multimodal Agricultural Agent Architecture (MA3): A New Paradigm for Intelligent Agricultural Decision-Making

TL;DR

MA3 introduces a unified, multimodal agricultural agent architecture that combines cross-modal perception, a lightweight router for tool selection, and an expert VQA model to enable accurate classification, robust detection, and intelligent decision support for sugarcane cultivation. It builds a five-task dataset (classification, detection, tool selection, VQA, agent evaluation), trains dedicated components (SDC, SDOD, and a Router), and validates performance with a multi-dimensional evaluation framework, demonstrating practical applicability and robustness. The approach addresses limitations of LLM-driven tool selection by using a supervised Router to reduce latency and hallucination risk, while enabling scalable extension to additional crops and tasks. The work provides open-source datasets and tools to advance domain-specific multimodal agricultural AI systems with real-world impact on crop health management and yield optimization.

Abstract

As a strategic pillar industry for human survival and development, modern agriculture faces dual challenges: optimizing production efficiency and achieving sustainable development. Against the backdrop of intensified climate change leading to frequent extreme weather events, the uncertainty risks in agricultural production systems are increasing exponentially. To address these challenges, this study proposes an innovative \textbf{M}ultimodal \textbf{A}gricultural \textbf{A}gent \textbf{A}rchitecture (\textbf{MA3}), which leverages cross-modal information fusion and task collaboration mechanisms to achieve intelligent agricultural decision-making. This study constructs a multimodal agricultural agent dataset encompassing five major tasks: classification, detection, Visual Question Answering (VQA), tool selection, and agent evaluation. We propose a unified backbone for sugarcane disease classification and detection tools, as well as a sugarcane disease expert model. By integrating an innovative tool selection module, we develop a multimodal agricultural agent capable of effectively performing tasks in classification, detection, and VQA. Furthermore, we introduce a multi-dimensional quantitative evaluation framework and conduct a comprehensive assessment of the entire architecture over our evaluation dataset, thereby verifying the practicality and robustness of MA3 in agricultural scenarios. This study provides new insights and methodologies for the development of agricultural agents, holding significant theoretical and practical implications. Our source code and dataset will be made publicly available upon acceptance.

Figures (11)

• Figure 1: MA3 is a unified multimodal agricultural agent architecture for intelligent agricultural decision-making, supporting multiple tasks, including disease classification, disease detection, and visual question-answering, through a multi-stage pipeline: (a) task routing, (b) tool execution, (c) response organization and (d) evaluation.
• Figure 2: Multimodal Agricultural Agent Architecture (MA3). The MA3 architecture employs a Router to dynamically select among classification tools, detection tools, and the expert model, integrating their outputs with the input text and image before feeding them into the LLM.
• Figure 3: Examples of images of 18 sugarcane diseases.
• Figure 4: Prediction results for healthy sugarcane and diseased sugarcane.
• Figure 5: VQA data construction pipeline.