Large Language Models are Zero Shot Hypothesis Proposers

TL;DR

This paper investigates whether large language models can generate scientific hypotheses in zero-shot settings using a temporally partitioned biomedical literature dataset. It introduces a four-role multi-agent collaboration framework and leverages external tools to enhance hypothesis generation, accompanied by a bespoke evaluation scheme including four high-signal metrics and human validation. Key findings show LLMs can propose untrained yet verifiable hypotheses, with uncertainty and collaborative agent dynamics enhancing zero-shot generalization, while domain adaptation and prompt strategies exert nuanced effects. The work highlights the potential of LLMs as catalysts for discovery and suggests directions for refining evaluation metrics and uncertainty-driven approaches.

Abstract

Significant scientific discoveries have driven the progress of human civilisation. The explosion of scientific literature and data has created information barriers across disciplines that have slowed the pace of scientific discovery. Large Language Models (LLMs) hold a wealth of global and interdisciplinary knowledge that promises to break down these information barriers and foster a new wave of scientific discovery. However, the potential of LLMs for scientific discovery has not been formally explored. In this paper, we start from investigating whether LLMs can propose scientific hypotheses. To this end, we construct a dataset consist of background knowledge and hypothesis pairs from biomedical literature. The dataset is divided into training, seen, and unseen test sets based on the publication date to control visibility. We subsequently evaluate the hypothesis generation capabilities of various top-tier instructed models in zero-shot, few-shot, and fine-tuning settings, including both closed and open-source LLMs. Additionally, we introduce an LLM-based multi-agent cooperative framework with different role designs and external tools to enhance the capabilities related to generating hypotheses. We also design four metrics through a comprehensive review to evaluate the generated hypotheses for both ChatGPT-based and human evaluations. Through experiments and analyses, we arrive at the following findings: 1) LLMs surprisingly generate untrained yet validated hypotheses from testing literature. 2) Increasing uncertainty facilitates candidate generation, potentially enhancing zero-shot hypothesis generation capabilities. These findings strongly support the potential of LLMs as catalysts for new scientific discoveries and guide further exploration.

Figures (5)

• Figure 1: Illustrating an generated hypothesis utilizing the fine-tuned 65B LLaMA within our constructed datasets, which closely match the findings in existing literature.
• Figure 2: The iterative experimental loop of scientific discovery: observations and data accumulated from past experiments are analyzed and used to generate new hypotheses, and in turn new experiments that will yield new data to continue to cycle. In this paper, we mainly focus on investigating whether LLMs have the zero shot generalization ability to generate new hypotheses.
• Figure 3: Data partition pipeline.
• Figure 4: The conceptual system of multi-agent collaboration for hypothesis generation. The overall prototyping process is illustrated below, allowing users to choose optional involvement. We offer core role descriptions of multi-agents and the fully automated system above.
• Figure 5: Distribution of the background and hypothesis pairs (BHP) dataset: In the left panel, we present the publication distribution by year for the training and seen test datasets, indicating a steady increase year by year until January 2023. In the center panel, we depict the publication distribution by month for the unseen test dataset, which was sampled from August 2023 and emphasizes the latter part of the month. The right panel displays the distribution of keywords in abstracts from the training, seen test, and unseen test datasets, represented by blue, yellow, and red, respectively.