Internal Incoherency Scores for Constraint-based Causal Discovery Algorithms
Sofia Faltenbacher, Jonas Wahl, Rebecca Herman, Jakob Runge
TL;DR
This paper tackles the challenge that constraint-based causal discovery methods rely on strong mechanistic and statistical assumptions that are often untested in real data. It introduces internal coherency scores that quantify how well a PC-like method’s output aligns with the observed CI tests without requiring ground-truth graphs, and proves these scores detect all errors that are theoretically detectable from a single run. The authors categorize error sources into faithfulness, structural, CI-test, and finite-sample violations, and describe how orientation conflicts and ambiguities manifest as incoherencies that the scores can quantify. Through toy models, simulations, and a real-world Auto MPG dataset, they demonstrate that coherency scores provide actionable diagnostics for model plausibility and robustness, advocating their integration into standard causal-discovery workflows for better reliability and method selection.
Abstract
Causal discovery aims to infer causal graphs from observational or experimental data. Methods such as the popular PC algorithm are based on conditional independence testing and utilize enabling assumptions, such as the faithfulness assumption, for their inferences. In practice, these assumptions, as well as the functional assumptions inherited from the chosen conditional independence test, are typically taken as a given and not further tested for their validity on the data. In this work, we propose internal coherency scores that allow testing for assumption violations and finite sample errors, whenever detectable without requiring ground truth or further statistical tests. We provide a complete classification of erroneous results, including a distinction between detectable and undetectable errors, and prove that the detectable erroneous results can be measured by our scores. We illustrate our coherency scores on the PC algorithm with simulated and real-world datasets, and envision that testing for internal coherency can become a standard tool in applying constraint-based methods, much like a suite of tests is used to validate the assumptions of classical regression analysis.