Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
Martin Bertran, Riccardo Fogliato, Zhiwei Steven Wu
TL;DR
The paper addresses how empirical conclusions hinge on analytic decisions by demonstrating that fully autonomous AI analysts can reproduce the analytic diversity seen in human multi-analyst studies. By assigning pre-specified hypotheses to fixed datasets and varying underlying LLMs and prompt framings, the authors reveal broad dispersion in effect sizes, $p$-values, and binary conclusions, even after method validity screening. The key contributions include a scalable, auditable framework for generating and evaluating a large multiverse of analyses, decomposition of dispersion into model- and persona-driven components, and evidence that conclusions are steerable by analytic framing. This work has practical significance for metascience and data governance, highlighting the need to treat analysis outputs as distributions rather than single outcomes and to implement automated auditing to curb selective reporting in data-driven decision-making.
Abstract
The conclusions of empirical research depend not only on data but on a sequence of analytic decisions that published results seldom make explicit. Past ``many-analyst" studies have demonstrated this: independent teams testing the same hypothesis on the same dataset regularly reach conflicting conclusions. But such studies require months of coordination among dozens of research groups and are therefore rarely conducted. In this work, we show that fully autonomous AI analysts built on large language models (LLMs) can reproduce a similar structured analytic diversity cheaply and at scale. We task these AI analysts with testing a pre-specified hypothesis on a fixed dataset, varying the underlying model and prompt framing across replicate runs. Each AI analyst independently constructs and executes a full analysis pipeline; an AI auditor then screens each run for methodological validity. Across three datasets spanning experimental and observational designs, AI analyst-produced analyses display wide dispersion in effect sizes, $p$-values, and binary decisions on supporting the hypothesis or not, frequently reversing whether a hypothesis is judged supported. This dispersion is structured: recognizable analytic choices in preprocessing, model specification, and inference differ systematically across LLM and persona conditions. Critically, the effects are \emph{steerable}: reassigning the analyst persona or LLM shifts the distribution of outcomes even after excluding methodologically deficient runs.