Dynamic Expert-Guided Model Averaging for Causal Discovery

TL;DR

The paper tackles the challenge of choosing among diverse causal-discovery methods by introducing a dynamic Expert Model Averaging framework that operates on a set of component graphs $M$ and leverages an expert $e$ to approve or reject potential edges and determine their orientation via thresholds $ heta_1$ and $ heta_2$. Edges are added in a cycle-free manner, guided by majority signals across $M$ and by expert input when ambiguous, with a clear separation between edge existence and orientation decisions. Experiments on six Bayesys networks and the SimSUM dataset, across clean and noisy data, show that this ensemble approach improves BSF and F1 scores and often reduces SHD compared to static baselines, while remaining functional when the expert is imperfect (including simulated and LLM-based experts). The work demonstrates the practical viability of using dynamic expert knowledge to mediate model averaging in causal discovery and provides actionable insights for deployment, along with acknowledged limitations and directions for future work in diverse data regimes and clinical settings.

Abstract

Understanding causal relationships is critical for healthcare. Accurate causal models provide a means to enhance the interpretability of predictive models, and furthermore a basis for counterfactual and interventional reasoning and the estimation of treatment effects. However, would-be practitioners of causal discovery face a dizzying array of algorithms without a clear best choice. This abundance of competitive algorithms makes ensembling a natural choice for practical applications. At the same time, real-world use cases frequently face challenges that violate the assumptions of common causal discovery algorithms, forcing heavy reliance on expert knowledge. Inspired by recent work on dynamically requested expert knowledge and LLMs as experts, we present a flexible model averaging method leveraging dynamically requested expert knowledge to ensemble a diverse array of causal discovery algorithms. Experiments demonstrate the efficacy of our method with imperfect experts such as LLMs on both clean and noisy data. We also analyze the impact of different degrees of expert correctness and assess the capabilities of LLMs for clinical causal discovery, providing valuable insights for practitioners.

Figures (11)

• Figure 1: The mean delta in graphical accuracy metrics compared to our method (edge threshold $\theta_1 = 0.0$, orientation threshold $\theta_2 =0.7$, and expert correctness $=80\%$). To standardize the SHD, we compute the relative SHD to our method. $*$ indicates some missing results for an algorithm due to timeouts ($>$6h) or exceptions. $\dagger$ indicates some invalid results for an algorithm (only applicable to precision). Missing and invalid results are excluded from the calculation.
• Figure 2: Graphical accuracy metrics, averaged across all runs, plotted over expert correctnesses $\in \{50\%, 60\%, 70\%, 80\%, 90\%, 100\%\}$. Bayesys Model Avg is provided as a static baseline for comparison, as an ensembling method not using expert knowledge.
• Figure 3: Graphical accuracy metrics, averaged across all runs, plotted over edge threshold $\theta_1 \in \{0.0, 0.1, 0.2, ..., 1.0\}$, with fixed orientation threshold $\theta_2=0.7$ and expert correctness $80\%$. Bayesys Model Avg is provided as a static baseline for comparison.
• Figure 4: BSF on clean data, plotted over expert correctnesses $\in \{50\%, 60\%, 70\%, 80\%, 90\%, 100\%\}$. Bayesys Model Avg is provided as a static baseline for comparison, as an ensembling method not using expert knowledge. The shaded region denotes one standard deviation.
• Figure 5: BSF on noisy data, plotted over expert correctnesses $\in \{50\%, 60\%, 70\%, 80\%, 90\%, 100\%\}$. Bayesys Model Avg is provided as a static baseline for comparison, as an ensembling method not using expert knowledge. The shaded region denotes one standard deviation.