BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

TL;DR

BioAgent Bench tackles the challenge of realistically evaluating AI agents in bioinformatics by encoding end-to-end pipelines with explicit artifact outputs and stress-testing through perturbations. The approach benchmarks frontier closed-source and open-weight models across ten tasks using an LLM-based grader to adjudicate progress and results, treating model+harness as the agent. Key contributions include a public benchmark dataset, an evaluation harness that records traces and evaluates robustness, and practical insights into the stability of pipeline execution versus plan quality. The work has practical significance for enabling private, locally deployable agentic workflows in biomedical research and for guiding future improvements in planning, tool use, and error handling under real-world constraints.

Abstract

This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly.

Figures (3)

• Figure 1: An overview of BioAgent Bench. Inputs to LLM agents consist of a task prompt, input data, and reference data. While solving the provided task, an agent can use general-purpose packages or specialized bioinformatics tools. After the agent finishes generation, LLM judge compares its outputs against ground truth and produces evaluation results. In addition to the standard "vanilla" inputs, we also experiment with different perturbations to stress-test the agents. We focus our evaluation on 10 tasks in bioinformatics (each designed around a different organism, virus, or an entire ecosystem), and 10 models (5 open-weight and 5 closed-weight models).
• Figure 2: Model-task completion heatmap. The left panel shows a pairwise completion matrix: rows and columns correspond to models and tasks, respectively, and each cell reports the completion rate (in %) for each model and task pair. Cell color encodes the completion rate, with numeric annotations shown for readability. The right panel summarizes performance across tasks by reporting each model’s average completion rate, providing an overall ranking of models.
• Figure 3: Scatter plot comparing each model’s average plan quality score against its overall pipeline completion rate