BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation

TL;DR

BioVerge addresses the need for standardized benchmarks and execution environments in biomedical hypothesis generation by unifying structured triplets, PubMed literature, and a knowledge graph. It introduces BioVerge Agent, a ReAct-based framework with Generation and Evaluation modules, and compares Single and Double memory architectures across multiple data sources and evaluation thresholds. Experimental results show that leveraging diverse data sources and iterative self-evaluation improves novelty and relevance of hypotheses, with distinct tradeoffs between exploration and precision. The benchmark and agent design aim to accelerate scientifically plausible hypotheses while enabling rigorous, tool-augmented discovery in biomedicine.

Abstract

Hypothesis generation in biomedical research has traditionally centered on uncovering hidden relationships within vast scientific literature, often using methods like Literature-Based Discovery (LBD). Despite progress, current approaches typically depend on single data types or predefined extraction patterns, which restricts the discovery of novel and complex connections. Recent advances in Large Language Model (LLM) agents show significant potential, with capabilities in information retrieval, reasoning, and generation. However, their application to biomedical hypothesis generation has been limited by the absence of standardized datasets and execution environments. To address this, we introduce BioVerge, a comprehensive benchmark, and BioVerge Agent, an LLM-based agent framework, to create a standardized environment for exploring biomedical hypothesis generation at the frontier of existing scientific knowledge. Our dataset includes structured and textual data derived from historical biomedical hypotheses and PubMed literature, organized to support exploration by LLM agents. BioVerge Agent utilizes a ReAct-based approach with distinct Generation and Evaluation modules that iteratively produce and self-assess hypothesis proposals. Through extensive experimentation, we uncover key insights: 1) different architectures of BioVerge Agent influence exploration diversity and reasoning strategies; 2) structured and textual information sources each provide unique, critical contexts that enhance hypothesis generation; and 3) self-evaluation significantly improves the novelty and relevance of proposed hypotheses.

Figures (9)

• Figure 1: The ABC principle as a method to explore Literature Based Discovery (LBD).
• Figure 2: The cutoff date separation between the historical knowledge base and test dataset candidates for dataset construction. The knowledge base contains article corpus, knowledge graph, and triplet representations. The test dataset is cleaned to ensure all test queries are proposed after Jan. 1, 2024, related to Diabetes Mellitus, and ranked by the impact factor (IF) score of its publishing journal.
• Figure 3: Generation and Evaluation modules' ReAct workflow. Given a query, the agent iteratively reflects on its current state, executes an action, and stores the observed output until termination.
• Figure 4: Single Agent (left) and Double Agent (right) architectures. Memory is shared between Generation and Evaluation modules in Single Agent and separated in Double Agent.
• Figure 5: BioVerge Agent execution example. In Generation module the agent gathers information via APIs and proposes a relation and hypothesis description. Then in Evaluation module the agent checks novelty, verifies reasoning trajectories, and outputs feedback for the hypothesis proposal.