Can Large Language Models Help Experimental Design for Causal Discovery?

TL;DR

The paper addresses how to design informative interventions for causal discovery when interventional data are scarce. It introduces LeGIT, a framework that warms up online causal discovery with an LLM to inject world knowledge about experimental design and then relies on a numerical method (e.g., GIT/ENCO) to select subsequent targets. Empirically, LeGIT achieves state-of-the-art performance on four real-world benchmark graphs and can rival or surpass human strategies, especially in complex datasets and low-data regimes. A theoretical discussion provides convergence guarantees under standard identifiability assumptions, illustrating that LeGIT accelerates convergence by reducing ambiguities in the early search space.

Abstract

Designing proper experiments and selecting optimal intervention targets is a longstanding problem in scientific or causal discovery. Identifying the underlying causal structure from observational data alone is inherently difficult. Obtaining interventional data, on the other hand, is crucial to causal discovery, yet it is usually expensive and time-consuming to gather sufficient interventional data to facilitate causal discovery. Previous approaches commonly utilize uncertainty or gradient signals to determine the intervention targets. However, numerical-based approaches may yield suboptimal results due to the inaccurate estimation of the guiding signals at the beginning when with limited interventional data. In this work, we investigate a different approach, whether we can leverage Large Language Models (LLMs) to assist with the intervention targeting in causal discovery by making use of the rich world knowledge about the experimental design in LLMs. Specifically, we present Large Language Model Guided Intervention Targeting (LeGIT) -- a robust framework that effectively incorporates LLMs to augment existing numerical approaches for the intervention targeting in causal discovery. Across 4 realistic benchmark scales, LeGIT demonstrates significant improvements and robustness over existing methods and even surpasses humans, which demonstrates the usefulness of LLMs in assisting with experimental design for scientific discovery.

Figures (24)

• Figure 1: Illustration of the LeGIT framework. The left side represents the loop of Online Causal Discovery, while the right side illustrates the experiment design process. In Step (a), Large Language Models (LLMs) warm up the causal discovery process by leveraging world knowledge and aligning it with the experiment's meta-information. This enables the identification of clear causal structures, which, in Step (b), guide previous methods to pinpoint informative intervention targets effectively.
• Figure 2: At the initial stage of the online causal discovery, the intervention targets from LLM-based selection and gradient-based selection.
• Figure 3: Prompt template at warmup stage.
• Figure 4: SHD metric for different methods (over 5 seeds) towards different intervention samples. ($T = 33$ rounds, $|D_{int}^I| = 32, N = 1056$)
• Figure 5: The selected Node Frequency obtained by different strategies on Epoch 0-4 from 5 different seeds under Table\ref{['table:shd_and_sid']} setting.

Theorems & Definitions (5)

• Lemma C.1: LLM Warmup is Sufficiently Informative
• proof
• Proposition C.2: Convergence of LeGIT
• proof : Proof
• Remark C.3: Extension to Other Methods