Generating Literature-Driven Scientific Theories at Scale

TL;DR

The paper addresses the challenge of synthesizing scientific theories directly from literature by proposing a scalable, literature-grounded theory-generation pipeline called Theorizer. It formalizes a problem where a user query and corpus yield a set of theories described by laws, scopes, and supporting evidence, evaluated against five desiderata. Through large-scale experiments, it demonstrates that literature-grounded theory generation yields significantly higher empirical support and predictive accuracy than parametric baselines, albeit at higher computational cost, and reveals a tradeoff between novelty and reliability. This work advances automated theory formation, offering a pragmatic path to scalable, literature-informed scientific insight with implications for SEO, AI-assisted research, and knowledge discovery.

Abstract

Contemporary automated scientific discovery has focused on agents for generating scientific experiments, while systems that perform higher-level scientific activities such as theory building remain underexplored. In this work, we formulate the problem of synthesizing theories consisting of qualitative and quantitative laws from large corpora of scientific literature. We study theory generation at scale, using 13.7k source papers to synthesize 2.9k theories, examining how generation using literature-grounding versus parametric knowledge, and accuracy-focused versus novelty-focused generation objectives change theory properties. Our experiments show that, compared to using parametric LLM memory for generation, our literature-supported method creates theories that are significantly better at both matching existing evidence and at predicting future results from 4.6k subsequently-written papers

Figures (3)

• Figure 1: An overview of synthesizing theories from scientific literature with Theorizer. A user-provided theory query guides a search for scientific papers, then theory-relevant knowledge is extracted from each paper. That knowledge is provided to a language model which generates and refines a set of theories. Full example theories are large and provided in the Appendix.
• Figure 2: An overview of the predictive accuracy evaluation procedure. For each generated theory law, a language model is used to generate a detailed list of predictions. PaperFinder is used to find papers that may speak to those predictions, and each paper is rated as supporting, contradicting, or having no evidence for each prediction. This evidence is tallied across papers to arrive at final estimates of predictive precision and recall for a given law.
• Figure 3: Monte Carlo analysis of theory law overlap when repeatedly generating theories using the same theory query. Parametric and literature-supported series measure duplicates within group (i.e. randomly select a parametric theory, then check whether it is duplicated in a random sample of N parametric theories). The literature-supported vs parametric series measures duplicates across groups (i.e. randomly select a parametric theory, then check wither it is duplicated in a random sample of N literature-supported theories), and is designed to measure whether literature-supported vs parametric theories are generating similar content. Duplication is measured using a pairwise LLM-as-a-judge using two input laws. Each point reflects the average of 50 samples. Shaded areas represent standard deviation across the 50 samples. Additional details in the Appendix.