HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation

TL;DR

HypoBench addresses the lack of principled evaluation for hypothesis generation by formalizing what constitutes a good hypothesis and building a benchmark that combines real-world and synthetic datasets with a multi-dimensional evaluation including explanatory power, practical utility, and generalizability. It systematically compares multiple LLMs and hypothesis-generation methods, showing data-driven approaches outperform baselines on real data and revealing remaining gaps in synthetic scenarios as task difficulty increases. The study introduces novel evaluation metrics such as hypothesis discovery rate and assesses qualitative properties like novelty, plausibility, and clarity, providing actionable insights into balancing plausibility and novelty. Overall, HypoBench offers a valuable resource for advancing AI-assisted scientific discovery and identifies clear directions for improving hypothesis generation methods.

Abstract

There is growing interest in hypothesis generation with large language models (LLMs). However, fundamental questions remain: what makes a good hypothesis, and how can we systematically evaluate methods for hypothesis generation? To address this, we introduce HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets. We evaluate four state-of-the-art LLMs combined with six existing hypothesis-generation methods. Overall, our results suggest that existing methods are capable of discovering valid and novel patterns in the data. However, the results from synthetic datasets indicate that there is still significant room for improvement, as current hypothesis generation methods do not fully uncover all relevant or meaningful patterns. Specifically, in synthetic settings, as task difficulty increases, performance significantly drops, with best models and methods only recovering 38.8% of the ground-truth hypotheses. These findings highlight challenges in hypothesis generation and demonstrate that HypoBench serves as a valuable resource for improving AI systems designed to assist scientific discovery.

Figures (6)

• Figure 1: An overview of our benchmark. We curate 194 datasets spanning 7 real-world and 5 synthetic domains. We illustrate how difficulty levels are controlled in our synthetic settings by showing an example from the college admission task. Our evaluation measures explanatory power and interestingness of generate hypotheses.
• Figure 2: HypoGeniC hypothesis discovery rate (HDR) results on synthetic datasets with different task difficulty. As task difficulty increases, HDR substantially drops, even to below 30% sometimes.
• Figure 3: HDR scores of Zero-shot Generation and HypoGeniC on four different synthetic datasets: Presidential Election, Personality Prediction, College Admission, and Shoe Sales. The results show that model priors can affect the quality of the generated hypotheses in different datasets.
• Figure 4: HypoGeniC HDR scores on the College Admission datasets under different difficulty controlls. Top: normal ground-truth hypotheses; bottom: counterintuitive ground-truth hypotheses.
• Figure 5: HypoGeniC F1 scores on synthetic datasets with different task difficulty.