SeedBench: A Multi-task Benchmark for Evaluating Large Language Models in Seed Science

TL;DR

SeedBench addresses the lack of domain-specific evaluation benchmarks in seed science by introducing a first-of-its-kind multi-task benchmark for seed breeding. It models the seed-breeding workflow into three tasks and 11 subcategories, built from a large literature corpus and expert-validated questions, and evaluates 26 LLMs with zero-shot and one-shot prompts. The findings show that general-purpose models like DeepSeek-V3-671B and GPT-4 outperform domain-specific fine-tuned models, with an optimal model size around 7–14B and significant influence from prompt design. This work provides a concrete foundation for AI-assisted seed design, highlights gaps between current LLM capabilities and breeding needs, and points to future directions in knowledge integration, multimodal data, and safe/transparent AI for agriculture.

Abstract

Seed science is essential for modern agriculture, directly influencing crop yields and global food security. However, challenges such as interdisciplinary complexity and high costs with limited returns hinder progress, leading to a shortage of experts and insufficient technological support. While large language models (LLMs) have shown promise across various fields, their application in seed science remains limited due to the scarcity of digital resources, complex gene-trait relationships, and the lack of standardized benchmarks. To address this gap, we introduce SeedBench -- the first multi-task benchmark specifically designed for seed science. Developed in collaboration with domain experts, SeedBench focuses on seed breeding and simulates key aspects of modern breeding processes. We conduct a comprehensive evaluation of 26 leading LLMs, encompassing proprietary, open-source, and domain-specific fine-tuned models. Our findings not only highlight the substantial gaps between the power of LLMs and the real-world seed science problems, but also make a foundational step for research on LLMs for seed design.

Figures (39)

• Figure 1: Breeding Expert Workflow Framework. We establish benchmark construction principles by consulting domain experts to replicate real-world seed breeding decision-making processes. (1) Gene Information Retrieval, utilizing established databases to obtain gene sequences and expression patterns; (2) Gene Function & Regulation, employing experimental approaches (e.g., gene knockout, overexpression) to investigate gene roles in plant development; and (3) Variety Breeding & Trait Optimization, implementing breeding techniques (e.g., hybridization, backcrossing) combined with agronomic trait selection for stable variety development.
• Figure 2: Benchmark Taxonomy Distribution. Three core breeding steps are further divided into ten expert-curated subcategories within SeedBench, which comprises a total of 2,264 questions. The percentages shown in the diagram represent the proportion of questions in each category relative to 2,264.
• Figure 3: Benchmark Construction Pipeline. We developed SeedBench by extracting 308,727 breeding-related papers from English and Chinese sources and converting them into a unified Markdown format. The data underwent rigorous cleaning, ultimately yielding a 1.1-billion-token corpus. From this, experts curated 279 high-quality text segments, spanning 10 breeding subcategories, for generating LLM-based Q&A tasks. Validation included both automated and expert reviews, removing low-quality entries and ensuring relevance. SeedBench offers 2,264 refined questions across 11 task types, enabling fine-grained evaluation of LLMs in seed breeding.
• Figure 4: Performance vs. Model Size. We empirically validate scaling laws in seed breeding tasks, showing a logarithmic correlation between model size and average scores. The optimal model size for breeding tasks lies between 7B and 14B, balancing performance and computational efficiency.
• Figure 5: An illustration of the Content, Example Question, Classification, and Reference fields.