Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering

Haodong Chen, Guido Zuccon, Teerapong Leelanupab

TL;DR

This work demonstrates that open-source LLMs can match or exceed the performance of proprietary models in genomic QA when combined with a modular, multi-agent architecture. By reproducing GeneGPT with open backbones and then introducing OpenBioLLM, the authors show that specialized roles for tool routing, query generation, and response validation yield strong results on GeneTuring and GeneHop while reducing latency. Key contributions include a fully open, reproducible implementation, a comprehensive error analysis, and empirical guidance on model sizing and prompt design for multi-agent QA. The approach offers practical benefits in cost, privacy, scalability, and transparency, with broad implications for genomic data interfaces and agent-based IR/NLP systems.

Abstract

Genomic question answering often requires complex reasoning and integration across diverse biomedical sources. GeneGPT addressed this challenge by combining domain-specific APIs with OpenAI's code-davinci-002 large language model to enable natural language interaction with genomic databases. However, its reliance on a proprietary model limits scalability, increases operational costs, and raises concerns about data privacy and generalization. In this work, we revisit and reproduce GeneGPT in a pilot study using open source models, including Llama 3.1, Qwen2.5, and Qwen2.5 Coder, within a monolithic architecture; this allows us to identify the limitations of this approach. Building on this foundation, we then develop OpenBioLLM, a modular multi-agent framework that extends GeneGPT by introducing agent specialization for tool routing, query generation, and response validation. This enables coordinated reasoning and role-based task execution. OpenBioLLM matches or outperforms GeneGPT on over 90% of the benchmark tasks, achieving average scores of 0.849 on Gene-Turing and 0.830 on GeneHop, while using smaller open-source models without additional fine-tuning or tool-specific pretraining. OpenBioLLM's modular multi-agent design reduces latency by 40-50% across benchmark tasks, significantly improving efficiency without compromising model capability. The results of our comprehensive evaluation highlight the potential of open-source multi-agent systems for genomic question answering. Code and resources are available at https://github.com/ielab/OpenBioLLM.

Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering

TL;DR

This work demonstrates that open-source LLMs can match or exceed the performance of proprietary models in genomic QA when combined with a modular, multi-agent architecture. By reproducing GeneGPT with open backbones and then introducing OpenBioLLM, the authors show that specialized roles for tool routing, query generation, and response validation yield strong results on GeneTuring and GeneHop while reducing latency. Key contributions include a fully open, reproducible implementation, a comprehensive error analysis, and empirical guidance on model sizing and prompt design for multi-agent QA. The approach offers practical benefits in cost, privacy, scalability, and transparency, with broad implications for genomic data interfaces and agent-based IR/NLP systems.

Abstract

Genomic question answering often requires complex reasoning and integration across diverse biomedical sources. GeneGPT addressed this challenge by combining domain-specific APIs with OpenAI's code-davinci-002 large language model to enable natural language interaction with genomic databases. However, its reliance on a proprietary model limits scalability, increases operational costs, and raises concerns about data privacy and generalization. In this work, we revisit and reproduce GeneGPT in a pilot study using open source models, including Llama 3.1, Qwen2.5, and Qwen2.5 Coder, within a monolithic architecture; this allows us to identify the limitations of this approach. Building on this foundation, we then develop OpenBioLLM, a modular multi-agent framework that extends GeneGPT by introducing agent specialization for tool routing, query generation, and response validation. This enables coordinated reasoning and role-based task execution. OpenBioLLM matches or outperforms GeneGPT on over 90% of the benchmark tasks, achieving average scores of 0.849 on Gene-Turing and 0.830 on GeneHop, while using smaller open-source models without additional fine-tuning or tool-specific pretraining. OpenBioLLM's modular multi-agent design reduces latency by 40-50% across benchmark tasks, significantly improving efficiency without compromising model capability. The results of our comprehensive evaluation highlight the potential of open-source multi-agent systems for genomic question answering. Code and resources are available at https://github.com/ielab/OpenBioLLM.

Paper Structure

This paper contains 30 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction and Related Work
  2. Genomic QA Tasks
  3. GeneTuring
  4. GeneHop
  5. GeneGPT
  6. Pilot Study: Reproducing GeneGPT
  7. Open-source LLMs with GeneGPT
  8. Selecting model sizes for evaluation
  9. Experimenting with GeneGPT prompts
  10. Evaluation methods
  11. Effectiveness of GeneGPT with Open LLMs
  12. Failure patterns and opportunities
  13. Obs. 1: Exact match & case sensitivity
  14. Obs. 2: Relevant answer ranked too low
  15. Obs. 3: Query difficulty
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Inference algorithm in GeneGPT
  • Figure 2: Snippet of the prompt component used in OpenBioLLM to enforce standardized output format.
  • Figure 3: Multi-Agent Pipeline in OpenBioLLM Framework.
  • Figure 4: Latency per task and overall average latency (in seconds) for the monolithic architecture (Qwen2.5-32B backbone) versus the multi-agent architecture. For the multi-agent setup, we evaluate two model size combinations: 32B+32B and 32B+14B. Statistical significance is reported relative to the monolithic baseline: *** p<0.001, ** p<0.01, * p<0.05; n.s. indicates non-significant differences.