Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPT-4 as Evaluator: Evaluating Large Language Models on Pest Management in Agriculture

Shanglong Yang, Zhipeng Yuan, Shunbao Li, Ruoling Peng, Kang Liu, Po Yang

TL;DR

This work investigates whether large language models can generate useful pest-management advice for agriculture and introduces GPT-4 as a multi-dimensional evaluator of such output. By comparing GPT-3.5, GPT-4, and FLAN-T5, and by using an expert-system baseline for factual accuracy, the study explores zero-shot, few-shot, instruction-based, and self-consistency prompting to produce pest scenarios. The key contributions include an innovative evaluation framework with a weighted scoring scheme and empirical findings showing that instruction-based prompting significantly boosts accuracy (around 72%) and linguistic quality for agricultural advice, while FLAN models lag behind. The results demonstrate the feasibility of LLMs as decision-support tools in pest management, underscoring the importance of domain knowledge integration and prompting strategies, and they point to future work on refining prompts and providing stage-specific, actionable guidance for farmers.

Abstract

In the rapidly evolving field of artificial intelligence (AI), the application of large language models (LLMs) in agriculture, particularly in pest management, remains nascent. We aimed to prove the feasibility by evaluating the content of the pest management advice generated by LLMs, including the Generative Pre-trained Transformer (GPT) series from OpenAI and the FLAN series from Google. Considering the context-specific properties of agricultural advice, automatically measuring or quantifying the quality of text generated by LLMs becomes a significant challenge. We proposed an innovative approach, using GPT-4 as an evaluator, to score the generated content on Coherence, Logical Consistency, Fluency, Relevance, Comprehensibility, and Exhaustiveness. Additionally, we integrated an expert system based on crop threshold data as a baseline to obtain scores for Factual Accuracy on whether pests found in crop fields should take management action. Each model's score was weighted by percentage to obtain a final score. The results showed that GPT-3.4 and GPT-4 outperform the FLAN models in most evaluation categories. Furthermore, the use of instruction-based prompting containing domain-specific knowledge proved the feasibility of LLMs as an effective tool in agriculture, with an accuracy rate of 72%, demonstrating LLMs' effectiveness in providing pest management suggestions.

GPT-4 as Evaluator: Evaluating Large Language Models on Pest Management in Agriculture

TL;DR

This work investigates whether large language models can generate useful pest-management advice for agriculture and introduces GPT-4 as a multi-dimensional evaluator of such output. By comparing GPT-3.5, GPT-4, and FLAN-T5, and by using an expert-system baseline for factual accuracy, the study explores zero-shot, few-shot, instruction-based, and self-consistency prompting to produce pest scenarios. The key contributions include an innovative evaluation framework with a weighted scoring scheme and empirical findings showing that instruction-based prompting significantly boosts accuracy (around 72%) and linguistic quality for agricultural advice, while FLAN models lag behind. The results demonstrate the feasibility of LLMs as decision-support tools in pest management, underscoring the importance of domain knowledge integration and prompting strategies, and they point to future work on refining prompts and providing stage-specific, actionable guidance for farmers.

Abstract

In the rapidly evolving field of artificial intelligence (AI), the application of large language models (LLMs) in agriculture, particularly in pest management, remains nascent. We aimed to prove the feasibility by evaluating the content of the pest management advice generated by LLMs, including the Generative Pre-trained Transformer (GPT) series from OpenAI and the FLAN series from Google. Considering the context-specific properties of agricultural advice, automatically measuring or quantifying the quality of text generated by LLMs becomes a significant challenge. We proposed an innovative approach, using GPT-4 as an evaluator, to score the generated content on Coherence, Logical Consistency, Fluency, Relevance, Comprehensibility, and Exhaustiveness. Additionally, we integrated an expert system based on crop threshold data as a baseline to obtain scores for Factual Accuracy on whether pests found in crop fields should take management action. Each model's score was weighted by percentage to obtain a final score. The results showed that GPT-3.4 and GPT-4 outperform the FLAN models in most evaluation categories. Furthermore, the use of instruction-based prompting containing domain-specific knowledge proved the feasibility of LLMs as an effective tool in agriculture, with an accuracy rate of 72%, demonstrating LLMs' effectiveness in providing pest management suggestions.
Paper Structure (19 sections, 1 equation, 1 figure, 2 tables)

This paper contains 19 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Application of LLMs
  4. Prompt & Prompt Engineering
  5. Experiment Design
  6. Experiment Models
  7. GPT
  8. FLAN-T5
  9. Baselines
  10. Expert System
  11. Generation of Input Samples from Expert System
  12. Experiment Prompting
  13. Zero-shot Prompting
  14. Few-shot Prompting
  15. Instruction-based Prompting
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Generation of Input Samples from Expert System. This image outlines a process for generating labelled pest samples, detailing steps from data extraction and indexing of pests, through generating true and false density values and crops, to the creation of positive and negative samples for experiment.