Can LLMs Leverage Observational Data? Towards Data-Driven Causal Discovery with LLMs
Yuni Susanti, Michael Färber
TL;DR
The paper investigates data-driven causal discovery by enabling LLMs to process structured observational data within prompts. It formalizes causal graph inference as classifying pairs of variables into directional or non-relational edges and introduces two prompting schemes, Pairwise and BFS, augmented with a sampled observational dataset. Empirical results on BNLearn benchmarks show that incorporating observational data improves $F1$ scores, with BFS prompting delivering the strongest performance and surpassing traditional baselines by notable margins. These findings support a hybrid direction that integrates statistical causal discovery with LLM-based reasoning, while also outlining limitations and avenues for scaling to larger datasets and multiple models.
Abstract
Causal discovery traditionally relies on statistical methods applied to observational data, often requiring large datasets and assumptions about underlying causal structures. Recent advancements in Large Language Models (LLMs) have introduced new possibilities for causal discovery by providing domain expert knowledge. However, it remains unclear whether LLMs can effectively process observational data for causal discovery. In this work, we explore the potential of LLMs for data-driven causal discovery by integrating observational data for LLM-based reasoning. Specifically, we examine whether LLMs can effectively utilize observational data through two prompting strategies: pairwise prompting and breadth first search (BFS)-based prompting. In both approaches, we incorporate the observational data directly into the prompt to assess LLMs' ability to infer causal relationships from such data. Experiments on benchmark datasets show that incorporating observational data enhances causal discovery, boosting F1 scores by up to 0.11 point using both pairwise and BFS LLM-based prompting, while outperforming traditional statistical causal discovery baseline by up to 0.52 points. Our findings highlight the potential and limitations of LLMs for data-driven causal discovery, demonstrating their ability to move beyond textual metadata and effectively interpret and utilize observational data for more informed causal reasoning. Our studies lays the groundwork for future advancements toward fully LLM-driven causal discovery.